From 68f6a82dc18ac256279b7b8cd2b88a2642c90bee Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 11 Jun 2026 22:57:55 +0200 Subject: [PATCH 01/75] docs: design spec for zarrs-backed low-level functional API Co-Authored-By: Claude Fable 5 --- .../2026-06-11-zarrs-functional-api-design.md | 180 ++++++++++++++++++ 1 file changed, 180 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md diff --git a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md new file mode 100644 index 0000000000..ad28c987f8 --- /dev/null +++ b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md @@ -0,0 +1,180 @@ +# zarrs-backed low-level functional API for zarr-python + +Date: 2026-06-11 +Status: approved +Branch: `zarrs-bindings` + +## Goal + +Give zarr-python a low-level, functional API for zarr hierarchy CRUD whose +implementation delegates to the Rust [`zarrs`](https://docs.rs/zarrs) crate via +new PyO3 bindings. Every array routine takes a metadata document as an explicit +parameter, so callers can operate on read-only or virtual views of arrays +(e.g. decode a chunk with metadata the store never saw, or read a chunk as raw +bytes without decoding). + +Non-goals for this work: rewiring zarr-python's `Array`/`Group` classes or the +codec-pipeline registry through this API (possible later), fancy +(non-slice) indexing, and use of zarrs's experimental async feature. + +## Background + +- zarr-python is pure Python (hatchling). Its `Store` ABC + (`src/zarr/abc/store.py`) is async; metadata classes live under + `src/zarr/core/metadata/`. +- The Rust `zarrs` crate (~0.23) supports exactly the metadata-driven shape we + need: `Array::new_with_metadata(storage, path, metadata)` and + `Group::new_with_metadata(...)` construct nodes from a metadata document + without touching the store; `store_metadata()` persists separately. Chunk and + region I/O: `retrieve_chunk`, `retrieve_encoded_chunk` (raw bytes), + `retrieve_array_subset`, `partial_decoder` (sharding-aware), and the + corresponding `store_*` methods. `ArrayMetadata`/`GroupMetadata` parse + directly from JSON strings (v2 or v3; v2 converts internally). +- The existing `zarrs` PyPI package (github.com/zarrs/zarrs-python) exposes only + a codec pipeline (`CodecPipelineImpl`) and supports only a fixed set of + native stores. It cannot provide the API designed here, but its build setup + (maturin, PyO3 abi3, tokio/rayon) is the reference for ours. + +## Architecture + +Two distributions in this repo, hard boundary between them: + +1. **Rust crate `zarrs-bindings`** at the repo root (`zarrs-bindings/`), + built with maturin (PyO3, `abi3-py311`), publishing wheel `zarrs-bindings` + with native module `_zarrs_bindings`. It is a thin, mechanical binding over + `zarrs`: functions/pyclasses take metadata as a **JSON string**, a + store-config object, a node path, and return bytes / numpy arrays. It knows + nothing about zarr-python except the store sniffing described below. +2. **Python subpackage `zarr.zarrs`** in zarr-python: the public functional + API. Owns conversion between zarr-python types (`dict` metadata documents, + `zarr.abc.store.Store`, numpy arrays) and the binding layer, plus + validation, ergonomics, and error translation. Imports `_zarrs_bindings` + lazily and raises a helpful `ImportError` naming the `zarr[zarrs]` extra if + it is missing. + +zarr-python's own wheel remains pure Python; `zarrs-bindings` becomes an +optional dependency (`zarr[zarrs]`). + +## Public API (`zarr.zarrs`) + +All functions are `async def`. Parameters: + +- `metadata`: `dict[str, JSON]` — the literal metadata document (`zarr.json`, + or v2 `.zarray`/`.zgroup` equivalents). Never read from the store by the + array routines. +- `store`: `zarr.abc.store.Store`. +- `path`: node path within the store (str, `""` = root). +- `chunk_coords`: `tuple[int, ...]` grid coordinates. +- `selection`: tuple of `slice`/`int` only (v1 restriction). +- `options`: every function also accepts keyword-only + `options: ZarrsOptions | None = None` (omitted from the signatures below for + brevity) — a dataclass holding concurrency limits and checksum validation + flags. Defaults are applied when omitted; in Phase 1 the dataclass exists + but carries only defaults (fields become meaningful in Phase 3). + +```python +# node lifecycle +async def create_new_group(metadata, store, path) -> None # error if node exists +async def create_overwrite_group(metadata, store, path) -> None +async def create_new_array(metadata, store, path) -> None +async def create_overwrite_array(metadata, store, path) -> None +async def read_metadata(store, path) -> dict[str, JSON] # array or group doc +async def delete_node(store, path) -> None +async def list_children(store, path) -> list[tuple[str, dict]] # (path, metadata) + +# chunk-level I/O +async def decode_chunk(metadata, store, path, chunk_coords, *, selection=None) -> np.ndarray +async def read_encoded_chunk(metadata, store, path, chunk_coords) -> bytes | None +async def encode_chunk(metadata, store, path, chunk_coords, value) -> None +async def erase_chunk(metadata, store, path, chunk_coords) -> None + +# region-level I/O (selection in array coordinates, may span chunks) +async def decode_region(metadata, store, path, selection) -> np.ndarray +async def encode_region(metadata, store, path, selection, value) -> None +``` + +Mapping to zarrs primitives: + +| API function | zarrs primitive | +|---|---| +| `create_new_group` / `create_overwrite_group` | `Group::new_with_metadata` + `store_metadata` (existence check first for `new`) | +| `create_new_array` / `create_overwrite_array` | `Array::new_with_metadata` + `store_metadata` | +| `read_metadata` | `Array::open` / `Group::open` metadata retrieval | +| `delete_node` | `erase_metadata` + chunk erasure / prefix delete | +| `list_children` | `Group::children` / `traverse` | +| `decode_chunk` (no selection) | `retrieve_chunk` | +| `decode_chunk` (selection) | `partial_decoder(chunk).partial_decode` (sharding-aware) | +| `read_encoded_chunk` | `retrieve_encoded_chunk` | +| `encode_chunk` | `store_chunk` | +| `erase_chunk` | `erase_chunk` | +| `decode_region` | `retrieve_array_subset` | +| `encode_region` | `store_array_subset` | + +## Store bridge + +A Rust-side `StoreConfig` resolver, tried in priority order: + +1. `zarr.storage.LocalStore` → native `zarrs_filesystem` store. +2. obstore-backed `ObjectStore` → `zarrs_object_store` (Phase 3). +3. **Anything else** → generic `PyStore`: a Rust struct implementing + `ReadableStorageTraits` / `WritableStorageTraits` / + `ListableStorageTraits` over a Python callback object. + +The callback path: the async API function wraps the user's `Store` in a small +sync Python shim whose methods submit coroutines to zarr-python's existing +sync event-loop thread (`zarr.core.sync`, +`asyncio.run_coroutine_threadsafe(...)` + blocking result). Rust calls the +shim while holding no locks of its own. This makes any conformant `Store` +(Memory, Zip, Logging, Wrapper, user-defined) work without Rust knowing its +type. Deadlock safety relies on the existing invariant that code running on +the zarr sync loop never blocks on these Rust entry points. + +## Sync/async seam + +The public API is async to match zarr-python conventions. Internally each +function calls a blocking Rust entry point via `asyncio.to_thread`; the Rust +side releases the GIL during I/O and compute (reacquiring it only inside +`PyStore` callbacks). zarrs's experimental async feature is not used. + +## Error handling + +The binding layer raises a small set of typed exceptions defined in one place: +`NodeExistsError`, `NodeNotFoundError`, and `ValueError` subclasses for +metadata-parse and decode failures. `zarr.zarrs` translates to zarr-python +native exception types where an obvious equivalent exists (e.g. +`zarr.errors.ContainsArrayError`). Store-callback exceptions from Python +propagate through Rust unchanged. + +## Testing + +`tests/zarrs/`, module-level skip when `_zarrs_bindings` is not importable. + +- **Differential tests** are the core: every operation checked against + zarr-python's own implementation on the same store — write with zarr-python, + read with zarrs, and vice versa; metadata documents produced by both must + round-trip. +- Parametrized over: `MemoryStore` (exercises generic bridge) and `LocalStore` + (native path); zarr formats v2 and v3; a codec matrix including + `sharding_indexed`. +- Read-only-view tests: decode a chunk using a metadata dict not present in + the store; `read_encoded_chunk` returns bytes identical to `store.get`. +- A CI job builds the crate with `maturin develop` and runs `tests/zarrs/`. + Existing CI jobs are untouched (the suite skips without the extension). + +## Phasing + +1. **Phase 1**: crate scaffolding (maturin, CI build), store bridge (native + LocalStore + generic PyStore), node lifecycle functions, whole-chunk + `decode_chunk` / `read_encoded_chunk` / `encode_chunk` / `erase_chunk`. +2. **Phase 2**: `decode_region` / `encode_region`, chunk-subset `selection` + via partial decoders. +3. **Phase 3**: `ZarrsOptions` surface (concurrency, checksum validation, + direct IO), obstore native path, benchmarks vs. the pure-Python pipeline. + +## Naming decisions + +- Python API: `zarr.zarrs`. +- Rust crate / PyPI distribution: `zarrs-bindings` (PyPI name `zarrs` is taken + by the existing project); native module `_zarrs_bindings`. +- Function names follow the requested `create_new_*` / `create_overwrite_*` + pattern; reads are `decode_*` / `read_*`, writes `encode_*`. From 79c0bce73db09d73acbde329cbdd766e7fb4cdbf Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 08:16:03 +0200 Subject: [PATCH 02/75] docs: implementation plan for zarrs functional API phase 1 Co-Authored-By: Claude Fable 5 --- .../2026-06-12-zarrs-functional-api-phase1.md | 1796 +++++++++++++++++ 1 file changed, 1796 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-12-zarrs-functional-api-phase1.md diff --git a/docs/superpowers/plans/2026-06-12-zarrs-functional-api-phase1.md b/docs/superpowers/plans/2026-06-12-zarrs-functional-api-phase1.md new file mode 100644 index 0000000000..364de7cebc --- /dev/null +++ b/docs/superpowers/plans/2026-06-12-zarrs-functional-api-phase1.md @@ -0,0 +1,1796 @@ +# zarrs functional API (Phase 1) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A new in-repo PyO3 crate `zarrs-bindings` plus a `zarr.zarrs` subpackage exposing an async functional API (node lifecycle + whole-chunk I/O) that delegates to the Rust `zarrs` crate, working against any zarr-python `Store`. + +**Architecture:** Two layers. The Rust crate (`zarrs-bindings/`, maturin/PyO3 abi3-py312, native module `_zarrs_bindings`) is a thin binding over `zarrs` ≈0.23: functions take metadata as JSON strings, a store object, and a node path. The Python subpackage `src/zarr/zarrs/` owns the public API: dict metadata documents, `Store` adaptation (native `LocalStore` fast path + a generic sync-shim callback bridge), numpy conversion, and error translation. Spec: `docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md`. + +**Tech Stack:** Rust 1.91+ (1.96 installed), zarrs 0.23 (default features), pyo3 0.28 (abi3-py312), maturin build backend driven by uv (no maturin CLI needed), pytest with `asyncio_mode = "auto"`. + +--- + +## Environment notes (read first) + +- **Python/pytest/mypy always via `uv run`** (user preference). +- **Build/refresh the extension:** `uv sync --group zarrs --reinstall-package zarrs-bindings`. Plain `uv run --group zarrs ...` does NOT reliably rebuild after Rust edits — always re-sync with `--reinstall-package zarrs-bindings` after touching `zarrs-bindings/`. +- **Fast Rust feedback:** `cargo check --manifest-path zarrs-bindings/Cargo.toml` (compiles without packaging a wheel). +- Builds need network access (crates.io for cargo, PyPI for maturin). The Claude Code sandbox on this host fails at bwrap init, so run build commands with the sandbox disabled. +- Pre-commit hooks (ruff format/check, mypy, codespell) run on `git commit`. If a hook modifies files, `git add` the changes and commit again. +- The Rust snippets below were written against verified zarrs 0.23.13 / zarrs_storage 0.4.3 signatures. If `cargo check` reports a mismatch (most likely candidates: the exact signature of `zarrs::node::node_exists`, the re-export path of `store_set_partial_many`, or `TryInto for &NodePath`), check https://docs.rs/zarrs/latest — the primitives all exist; only spelling may need adjustment. +- Docstrings use **markdown** (mkdocs), single backticks — not RST. + +## File structure + +``` +zarrs-bindings/ # new Rust crate (own wheel: zarrs-bindings / _zarrs_bindings) + Cargo.toml + pyproject.toml # maturin backend + src/lib.rs # pymodule, exceptions, shared error helpers + src/store.rs # PyStore bridge + store resolution + src/node.rs # group/array creation, read_metadata, delete_node, list_children + src/chunk.rs # retrieve/store/erase chunk, retrieve_encoded_chunk +src/zarr/zarrs/ # new Python subpackage (public API) + __init__.py # import guard + re-exports + _bridge.py # StoreShim (sync adapter over async Store), resolve_store + _api.py # async functional API, numpy/JSON conversion, error translation +tests/zarrs/ # new test directory (skips when bindings missing) + __init__.py + conftest.py # store fixtures, array_metadata helper + test_bridge.py + test_node.py + test_chunk.py +.github/workflows/zarrs.yml # new CI job +pyproject.toml # modified: zarrs dependency group, uv source, sdist exclude +.gitignore # modified: zarrs-bindings/target/ +changes/+zarrs-bindings.feature.md +``` + +--- + +### Task 1: Rust crate scaffolding + uv wiring + +**Files:** +- Create: `zarrs-bindings/Cargo.toml` +- Create: `zarrs-bindings/pyproject.toml` +- Create: `zarrs-bindings/src/lib.rs` +- Modify: `pyproject.toml` (root) +- Modify: `.gitignore` + +- [ ] **Step 1: Create `zarrs-bindings/Cargo.toml`** + +```toml +[package] +name = "zarrs-bindings" +version = "0.1.0" +edition = "2024" +rust-version = "1.91" +publish = false + +[lib] +name = "_zarrs_bindings" +crate-type = ["cdylib"] + +[dependencies] +pyo3 = { version = "0.28", features = ["abi3-py312"] } +serde_json = "1" +zarrs = "0.23" + +[profile.release] +lto = "thin" +``` + +- [ ] **Step 2: Create `zarrs-bindings/pyproject.toml`** + +```toml +[build-system] +requires = ["maturin>=1.7,<2"] +build-backend = "maturin" + +[project] +name = "zarrs-bindings" +version = "0.1.0" +description = "PyO3 bindings to the zarrs Rust crate, consumed by zarr.zarrs" +requires-python = ">=3.12" +license = "MIT" + +[tool.maturin] +module-name = "_zarrs_bindings" +strip = true +``` + +- [ ] **Step 3: Create `zarrs-bindings/src/lib.rs`** (exceptions + version only for now) + +```rust +use pyo3::exceptions::{PyRuntimeError, PyValueError}; +use pyo3::prelude::*; + +pyo3::create_exception!( + _zarrs_bindings, + NodeExistsError, + PyValueError, + "A node already exists at the given path." +); +pyo3::create_exception!( + _zarrs_bindings, + NodeNotFoundError, + PyValueError, + "No node was found at the given path." +); + +pub(crate) fn runtime_err(err: impl std::fmt::Display) -> PyErr { + PyRuntimeError::new_err(err.to_string()) +} + +pub(crate) fn value_err(err: impl std::fmt::Display) -> PyErr { + PyValueError::new_err(err.to_string()) +} + +#[pyfunction] +fn version() -> &'static str { + env!("CARGO_PKG_VERSION") +} + +#[pymodule] +fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { + m.add("NodeExistsError", m.py().get_type::())?; + m.add("NodeNotFoundError", m.py().get_type::())?; + m.add_function(wrap_pyfunction!(version, m)?)?; + Ok(()) +} +``` + +- [ ] **Step 4: Wire into the root `pyproject.toml`** + +Add to the `[dependency-groups]` table (after the `dev` group): + +```toml +zarrs = [ + {include-group = "test"}, + "zarrs-bindings", +] +``` + +Add a new section at the end of the file: + +```toml +[tool.uv.sources] +zarrs-bindings = { path = "zarrs-bindings" } +``` + +Add `"/zarrs-bindings",` to the `exclude` list under `[tool.hatch.build.targets.sdist]`. + +- [ ] **Step 5: Add `zarrs-bindings/target/` to `.gitignore`** + +- [ ] **Step 6: Lock, build, smoke-test** + +Run: `cargo check --manifest-path zarrs-bindings/Cargo.toml` +Expected: compiles clean (first run downloads ~zarrs dependency tree). + +Run: `uv lock && uv sync --group zarrs` +Expected: lockfile updated; `zarrs-bindings` builds via maturin and installs. + +Run: `uv run --group zarrs python -c "import _zarrs_bindings as z; print(z.version())"` +Expected: `0.1.0` + +- [ ] **Step 7: Commit** (include `zarrs-bindings/Cargo.lock`, which the build created) + +```bash +git add zarrs-bindings .gitignore pyproject.toml uv.lock +git commit -m "feat: scaffold zarrs-bindings PyO3 crate" +``` + +--- + +### Task 2: `zarr.zarrs` package skeleton + test scaffolding + +**Files:** +- Create: `src/zarr/zarrs/__init__.py` +- Create: `tests/zarrs/__init__.py` (empty) +- Create: `tests/zarrs/conftest.py` +- Test: `tests/zarrs/test_api.py` + +- [ ] **Step 1: Write the failing test** — `tests/zarrs/test_api.py` + +```python +from __future__ import annotations + + +def test_import() -> None: + import zarr.zarrs + + assert isinstance(zarr.zarrs.__version__, str) +``` + +- [ ] **Step 2: Create `tests/zarrs/__init__.py`** (empty file) **and `tests/zarrs/conftest.py`** + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import pytest + +pytest.importorskip("_zarrs_bindings", reason="zarrs-bindings is not installed") + +import zarr +from zarr.storage import LocalStore, MemoryStore + +if TYPE_CHECKING: + from pathlib import Path + + from zarr.abc.store import Store + + +@pytest.fixture(params=["memory", "local"]) +async def store(request: pytest.FixtureRequest, tmp_path: Path) -> Store: + """A writable store: MemoryStore exercises the generic Python-callback bridge, + LocalStore exercises the native zarrs filesystem store.""" + if request.param == "memory": + return await MemoryStore.open() + return await LocalStore.open(root=tmp_path / "store") + + +def array_metadata(**kwargs: Any) -> dict[str, Any]: + """Build an array metadata document using zarr-python itself, so the + documents fed to zarrs always match what zarr-python would write.""" + params: dict[str, Any] = { + "shape": (8, 8), + "chunks": (4, 4), + "dtype": "uint16", + "zarr_format": 3, + } | kwargs + arr = zarr.create_array(store=MemoryStore(), **params) + doc = dict(arr.metadata.to_dict()) + if params["zarr_format"] == 2: + # v2 attributes live in .zattrs, not in the .zarray document + doc.pop("attributes", None) + return doc +``` + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `uv run --group zarrs pytest tests/zarrs -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'zarr.zarrs'` + +- [ ] **Step 4: Create `src/zarr/zarrs/__init__.py`** + +```python +""" +Low-level functional API for zarr hierarchies, backed by the Rust +[`zarrs`](https://zarrs.dev) crate. + +This subpackage is experimental. It requires the `zarrs-bindings` package +(in-repo Rust crate; install for development with `uv sync --group zarrs`). + +All array routines take an explicit metadata document (a `dict` matching the +`zarr.json` / `.zarray` document) rather than reading metadata from the store, +which makes read-only and virtual views possible. +""" + +try: + import _zarrs_bindings +except ImportError as e: + raise ImportError( + "zarr.zarrs requires the `zarrs-bindings` package, which is not installed. " + "It is built from the zarr-python repository: run `uv sync --group zarrs`." + ) from e + +__version__: str = _zarrs_bindings.version() + +__all__ = ["__version__"] +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `uv run --group zarrs pytest tests/zarrs -v` +Expected: 1 passed. Also verify the skip path works in the default env: `uv run pytest tests/zarrs -v` → all skipped/deselected with "zarrs-bindings is not installed" (the default group lacks the bindings). + +- [ ] **Step 6: Commit** + +```bash +git add src/zarr/zarrs tests/zarrs +git commit -m "feat: add zarr.zarrs package skeleton and test scaffolding" +``` + +--- + +### Task 3: StoreShim — sync bridge over async stores (pure Python, TDD) + +**Files:** +- Create: `src/zarr/zarrs/_bridge.py` +- Test: `tests/zarrs/test_bridge.py` + +- [ ] **Step 1: Write the failing tests** — `tests/zarrs/test_bridge.py` + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr.storage import LocalStore, MemoryStore +from zarr.zarrs._bridge import StoreShim, resolve_store + +if TYPE_CHECKING: + from pathlib import Path + + +def test_shim_get_set_delete() -> None: + shim = StoreShim(MemoryStore()) + assert shim.get("a/b") is None + shim.set("a/b", b"xyz") + assert shim.get("a/b") == b"xyz" + assert shim.get_range("a/b", 1, 1) == b"y" + assert shim.get_range("a/b", 1, None) == b"yz" + assert shim.get_suffix("a/b", 2) == b"yz" + assert shim.getsize("a/b") == 3 + assert shim.getsize("missing") is None + shim.delete("a/b") + assert shim.get("a/b") is None + + +def test_shim_listing() -> None: + shim = StoreShim(MemoryStore()) + shim.set("zarr.json", b"{}") + shim.set("a/zarr.json", b"{}") + shim.set("a/c/0/0", b"\x00") + assert shim.list() == ["a/c/0/0", "a/zarr.json", "zarr.json"] + assert shim.list_prefix("a/") == ["a/c/0/0", "a/zarr.json"] + assert shim.list_dir("a/") == (["a/zarr.json"], ["a/c/"]) + assert shim.list_dir("") == (["zarr.json"], ["a/"]) + assert shim.getsize_prefix("a/") == 3 + shim.delete_prefix("a/") + assert shim.list() == ["zarr.json"] + + +def test_resolve_store(tmp_path: Path) -> None: + local = LocalStore(tmp_path) + assert resolve_store(local) == {"filesystem": str(tmp_path)} + # read-only LocalStore must go through the shim so writes are rejected in Python + assert isinstance(resolve_store(LocalStore(tmp_path, read_only=True)), StoreShim) + assert isinstance(resolve_store(MemoryStore()), StoreShim) +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run --group zarrs pytest tests/zarrs/test_bridge.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'zarr.zarrs._bridge'` + +- [ ] **Step 3: Create `src/zarr/zarrs/_bridge.py`** + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr.abc.store import OffsetByteRequest, RangeByteRequest, SuffixByteRequest +from zarr.core.buffer.core import default_buffer_prototype +from zarr.core.sync import _collect_aiterator, sync +from zarr.storage import LocalStore + +if TYPE_CHECKING: + from zarr.abc.store import Store + + +class StoreShim: + """ + Synchronous adapter over an async `Store`, called from Rust worker threads. + + Each method blocks the calling thread by submitting a coroutine to the zarr + event-loop thread (`zarr.core.sync`). Methods must never be called from the + zarr event-loop thread itself; the Rust bindings only call them from + `asyncio.to_thread` worker threads. + """ + + def __init__(self, store: Store) -> None: + self._store = store + self._prototype = default_buffer_prototype() + + def get(self, key: str) -> bytes | None: + buf = sync(self._store.get(key, prototype=self._prototype)) + return None if buf is None else buf.to_bytes() + + def get_range(self, key: str, offset: int, length: int | None) -> bytes | None: + byte_range = ( + RangeByteRequest(offset, offset + length) + if length is not None + else OffsetByteRequest(offset) + ) + buf = sync(self._store.get(key, prototype=self._prototype, byte_range=byte_range)) + return None if buf is None else buf.to_bytes() + + def get_suffix(self, key: str, suffix: int) -> bytes | None: + buf = sync( + self._store.get(key, prototype=self._prototype, byte_range=SuffixByteRequest(suffix)) + ) + return None if buf is None else buf.to_bytes() + + def set(self, key: str, value: bytes) -> None: + sync(self._store.set(key, self._prototype.buffer.from_bytes(value))) + + def delete(self, key: str) -> None: + sync(self._store.delete(key)) + + def delete_prefix(self, prefix: str) -> None: + sync(self._store.delete_dir(prefix.rstrip("/"))) + + def getsize(self, key: str) -> int | None: + try: + return sync(self._store.getsize(key)) + except FileNotFoundError: + return None + + def getsize_prefix(self, prefix: str) -> int: + return sync(self._store.getsize_prefix(prefix.rstrip("/"))) + + def list(self) -> list[str]: + return sorted(sync(_collect_aiterator(self._store.list()))) + + def list_prefix(self, prefix: str) -> list[str]: + return sorted(sync(_collect_aiterator(self._store.list_prefix(prefix)))) + + def list_dir(self, prefix: str) -> tuple[list[str], list[str]]: + """Return `(keys, prefixes)` directly under `prefix`, as zarrs expects: + full keys, and child prefixes ending in `/`.""" + stripped = prefix.rstrip("/") + children = sorted(sync(_collect_aiterator(self._store.list_dir(stripped)))) + keys: list[str] = [] + prefixes: list[str] = [] + for child in children: + full = f"{stripped}/{child}" if stripped else child + if sync(self._store.exists(full)): + keys.append(full) + else: + prefixes.append(full + "/") + return keys, prefixes + + +def resolve_store(store: Store) -> StoreShim | dict[str, str]: + """ + Convert a zarr `Store` into the representation `_zarrs_bindings` expects: + a config dict for stores with a native Rust implementation, otherwise a + `StoreShim` that Rust calls back into. + """ + if isinstance(store, LocalStore) and not store.read_only: + return {"filesystem": str(store.root)} + return StoreShim(store) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `uv run --group zarrs pytest tests/zarrs/test_bridge.py -v` +Expected: 3 passed. (If `list_dir`/`delete_dir`/`getsize_prefix` choke on the stripped prefix, check the `Store` ABC docstrings in `src/zarr/abc/store.py:348-501` — these methods take prefixes without trailing slashes.) + +- [ ] **Step 5: Commit** + +```bash +git add src/zarr/zarrs/_bridge.py tests/zarrs/test_bridge.py +git commit -m "feat: sync store bridge for zarrs bindings" +``` + +--- + +### Task 4: Rust store bridge + group creation, end to end + +**Files:** +- Create: `zarrs-bindings/src/store.rs` +- Create: `zarrs-bindings/src/node.rs` +- Modify: `zarrs-bindings/src/lib.rs` +- Create: `src/zarr/zarrs/_api.py` +- Modify: `src/zarr/zarrs/__init__.py` +- Test: `tests/zarrs/test_node.py` + +- [ ] **Step 1: Write the failing tests** — `tests/zarrs/test_node.py` + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import pytest + +import zarr +from zarr.core.buffer.core import default_buffer_prototype +from zarr.zarrs import NodeExistsError, create_new_group, create_overwrite_group + +if TYPE_CHECKING: + from zarr.abc.store import Store + +GROUP_META: dict[str, Any] = { + "zarr_format": 3, + "node_type": "group", + "attributes": {"answer": 42}, +} + + +async def test_create_new_group(store: Store) -> None: + await create_new_group(GROUP_META, store, "foo") + group = zarr.open_group(store=store, path="foo", mode="r") + assert dict(group.attrs) == {"answer": 42} + + +async def test_create_new_group_at_root(store: Store) -> None: + await create_new_group(GROUP_META, store, "") + group = zarr.open_group(store=store, mode="r") + assert dict(group.attrs) == {"answer": 42} + + +async def test_create_new_group_existing_node(store: Store) -> None: + await create_new_group(GROUP_META, store, "foo") + with pytest.raises(NodeExistsError): + await create_new_group(GROUP_META, store, "foo") + + +async def test_create_overwrite_group(store: Store) -> None: + # an array and its chunks previously occupied the path; overwrite removes both + arr = zarr.create_array(store=store, name="foo", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + assert await store.exists("foo/c/0") + await create_overwrite_group(GROUP_META, store, "foo") + group = zarr.open_group(store=store, path="foo", mode="r") + assert dict(group.attrs) == {"answer": 42} + assert not await store.exists("foo/c/0") + assert await store.get("foo/zarr.json", prototype=default_buffer_prototype()) is not None +``` + +- [ ] **Step 2: Run tests to verify they fail** + +Run: `uv run --group zarrs pytest tests/zarrs/test_node.py -v` +Expected: FAIL with `ImportError: cannot import name 'NodeExistsError' from 'zarr.zarrs'` + +- [ ] **Step 3: Create `zarrs-bindings/src/store.rs`** + +```rust +use std::sync::Arc; + +use pyo3::exceptions::PyValueError; +use pyo3::prelude::*; +use pyo3::types::{PyBytes, PyDict}; +use zarrs::filesystem::FilesystemStore; +use zarrs::storage::{ + Bytes, ByteRange, ByteRangeIterator, ListableStorageTraits, MaybeBytes, MaybeBytesIterator, + OffsetBytesIterator, ReadableStorageTraits, ReadableWritableListableStorage, StorageError, + StoreKey, StoreKeys, StoreKeysPrefixes, StorePrefix, WritableStorageTraits, +}; + +/// A zarrs store backed by a Python `zarr.zarrs._bridge.StoreShim`. +/// +/// Every method attaches to the Python interpreter and calls the shim, which +/// blocks on the zarr event loop. Blocking waits in Python release the GIL, so +/// the loop thread can make progress while a Rust worker waits here. +pub(crate) struct PyStore(Py); + +fn py_err(err: PyErr) -> StorageError { + StorageError::Other(err.to_string()) +} + +fn invalid(err: impl std::fmt::Display) -> StorageError { + StorageError::Other(err.to_string()) +} + +impl PyStore { + fn get_with_range( + &self, + key: &StoreKey, + range: Option<&ByteRange>, + ) -> Result { + Python::attach(|py| { + let shim = self.0.bind(py); + let result = match range { + None => shim.call_method1("get", (key.as_str(),)), + Some(ByteRange::FromStart(offset, length)) => { + shim.call_method1("get_range", (key.as_str(), *offset, *length)) + } + Some(ByteRange::Suffix(suffix)) => { + shim.call_method1("get_suffix", (key.as_str(), *suffix)) + } + } + .map_err(py_err)?; + if result.is_none() { + Ok(None) + } else { + let bytes: Vec = result.extract().map_err(py_err)?; + Ok(Some(Bytes::from(bytes))) + } + }) + } +} + +impl ReadableStorageTraits for PyStore { + fn get(&self, key: &StoreKey) -> Result { + self.get_with_range(key, None) + } + + fn get_partial_many<'a>( + &'a self, + key: &StoreKey, + byte_ranges: ByteRangeIterator<'a>, + ) -> Result, StorageError> { + let mut out = Vec::new(); + for byte_range in byte_ranges { + match self.get_with_range(key, Some(&byte_range))? { + Some(bytes) => out.push(Ok(bytes)), + None => return Ok(None), + } + } + Ok(Some(Box::new(out.into_iter()))) + } + + fn size_key(&self, key: &StoreKey) -> Result, StorageError> { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("getsize", (key.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err) + }) + } + + fn supports_get_partial(&self) -> bool { + true + } +} + +impl WritableStorageTraits for PyStore { + fn set(&self, key: &StoreKey, value: Bytes) -> Result<(), StorageError> { + Python::attach(|py| { + let data = PyBytes::new(py, &value); + self.0 + .bind(py) + .call_method1("set", (key.as_str(), data)) + .map_err(py_err)?; + Ok(()) + }) + } + + fn set_partial_many( + &self, + key: &StoreKey, + offset_values: OffsetBytesIterator, + ) -> Result<(), StorageError> { + // read-modify-write fallback provided by zarrs + zarrs::storage::store_set_partial_many(self, key, offset_values) + } + + fn supports_set_partial(&self) -> bool { + false + } + + fn erase(&self, key: &StoreKey) -> Result<(), StorageError> { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("delete", (key.as_str(),)) + .map_err(py_err)?; + Ok(()) + }) + } + + fn erase_prefix(&self, prefix: &StorePrefix) -> Result<(), StorageError> { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("delete_prefix", (prefix.as_str(),)) + .map_err(py_err)?; + Ok(()) + }) + } +} + +impl ListableStorageTraits for PyStore { + fn list(&self) -> Result { + Python::attach(|py| { + let keys: Vec = self + .0 + .bind(py) + .call_method0("list") + .map_err(py_err)? + .extract() + .map_err(py_err)?; + keys.into_iter() + .map(|k| StoreKey::new(k).map_err(invalid)) + .collect() + }) + } + + fn list_prefix(&self, prefix: &StorePrefix) -> Result { + Python::attach(|py| { + let keys: Vec = self + .0 + .bind(py) + .call_method1("list_prefix", (prefix.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err)?; + keys.into_iter() + .map(|k| StoreKey::new(k).map_err(invalid)) + .collect() + }) + } + + fn list_dir(&self, prefix: &StorePrefix) -> Result { + Python::attach(|py| { + let (keys, prefixes): (Vec, Vec) = self + .0 + .bind(py) + .call_method1("list_dir", (prefix.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err)?; + let keys = keys + .into_iter() + .map(|k| StoreKey::new(k).map_err(invalid)) + .collect::, StorageError>>()?; + let prefixes = prefixes + .into_iter() + .map(|p| StorePrefix::new(p).map_err(invalid)) + .collect::, StorageError>>()?; + Ok(StoreKeysPrefixes::new(keys, prefixes)) + }) + } + + fn size_prefix(&self, prefix: &StorePrefix) -> Result { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("getsize_prefix", (prefix.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err) + }) + } +} + +/// Convert the Python-side store representation (`zarr.zarrs._bridge.resolve_store` +/// output) into a zarrs storage handle. +pub(crate) fn resolve_store(obj: &Bound<'_, PyAny>) -> PyResult { + if let Ok(config) = obj.downcast::() { + if let Some(root) = config.get_item("filesystem")? { + let root: String = root.extract()?; + let store = + FilesystemStore::new(root).map_err(|e| PyValueError::new_err(e.to_string()))?; + return Ok(Arc::new(store)); + } + return Err(PyValueError::new_err("unrecognized store configuration")); + } + Ok(Arc::new(PyStore(obj.clone().unbind()))) +} +``` + +- [ ] **Step 4: Create `zarrs-bindings/src/node.rs`** (group functions only; later tasks extend this file) + +```rust +use pyo3::prelude::*; +use zarrs::group::Group; +use zarrs::metadata::GroupMetadata; +use zarrs::node::{node_exists, NodePath}; +use zarrs::storage::{ReadableWritableListableStorage, StorePrefix}; + +use crate::store::resolve_store; +use crate::{runtime_err, value_err, NodeExistsError}; + +pub(crate) fn parse_node_path(path: &str) -> PyResult { + NodePath::new(path).map_err(value_err) +} + +/// When a node exists at `node_path`: erase it (and everything under it) if +/// `overwrite`, otherwise raise `NodeExistsError`. +pub(crate) fn prepare_target( + storage: &ReadableWritableListableStorage, + node_path: &NodePath, + overwrite: bool, +) -> PyResult<()> { + if node_exists(storage, node_path).map_err(runtime_err)? { + if !overwrite { + return Err(NodeExistsError::new_err(format!( + "a node already exists at path {}", + node_path.as_str() + ))); + } + let prefix: StorePrefix = node_path.try_into().map_err(value_err)?; + storage.erase_prefix(&prefix).map_err(runtime_err)?; + } + Ok(()) +} + +#[pyfunction] +pub(crate) fn create_group( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + overwrite: bool, +) -> PyResult<()> { + let storage = resolve_store(store)?; + let metadata = GroupMetadata::try_from(metadata_json.as_str()).map_err(value_err)?; + py.detach(move || { + let node_path = parse_node_path(&path)?; + prepare_target(&storage, &node_path, overwrite)?; + let group = Group::new_with_metadata(storage, &path, metadata).map_err(value_err)?; + group.store_metadata().map_err(runtime_err) + }) +} +``` + +- [ ] **Step 5: Register in `zarrs-bindings/src/lib.rs`** + +Add after the `use` lines: + +```rust +mod node; +mod store; +``` + +Add to the `#[pymodule]` body before `Ok(())`: + +```rust + m.add_function(wrap_pyfunction!(node::create_group, m)?)?; +``` + +- [ ] **Step 6: Compile** + +Run: `cargo check --manifest-path zarrs-bindings/Cargo.toml` +Expected: success. If `node_exists` or `try_into::()` signatures mismatch, fix per https://docs.rs/zarrs/latest/zarrs/node/ (the helpers exist; argument form may differ, e.g. `node_exists(&storage, &node_path)` vs a `&Arc` receiver). + +- [ ] **Step 7: Create `src/zarr/zarrs/_api.py`** + +```python +from __future__ import annotations + +import asyncio +import json +from contextlib import contextmanager +from dataclasses import dataclass +from typing import TYPE_CHECKING + +import _zarrs_bindings as _zb + +from zarr.errors import NodeNotFoundError +from zarr.zarrs._bridge import resolve_store + +if TYPE_CHECKING: + from collections.abc import Iterator, Mapping + + from zarr.abc.store import Store + from zarr.core.common import JSON + +NodeExistsError = _zb.NodeExistsError +"""Raised by `create_new_*` when a node already exists at the target path.""" + + +@dataclass(frozen=True, slots=True) +class ZarrsOptions: + """Options for zarrs-backed operations. + + Currently empty: fields (concurrency limits, checksum validation) arrive in + a later phase. Accepting it now keeps signatures stable. + """ + + +def _node_path(path: str) -> str: + """Convert a zarr-python node path (`""`, `"foo/bar"`) to a zarrs node path + (`"/"`, `"/foo/bar"`).""" + return f"/{path.strip('/')}" + + +@contextmanager +def _translate_errors() -> Iterator[None]: + try: + yield + except _zb.NodeNotFoundError as err: + raise NodeNotFoundError(str(err)) from err + + +async def create_new_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create a group at `path` from a group metadata document. + + Raises `NodeExistsError` if any node already exists at `path`. + """ + with _translate_errors(): + await asyncio.to_thread( + _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), False + ) + + +async def create_overwrite_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create a group at `path`, deleting any existing node (and its children) first.""" + with _translate_errors(): + await asyncio.to_thread( + _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), True + ) +``` + +- [ ] **Step 8: Re-export from `src/zarr/zarrs/__init__.py`** + +Replace the `__version__`/`__all__` lines at the end with: + +```python +__version__: str = _zarrs_bindings.version() + +from zarr.zarrs._api import ( + NodeExistsError, + ZarrsOptions, + create_new_group, + create_overwrite_group, +) + +__all__ = [ + "NodeExistsError", + "ZarrsOptions", + "__version__", + "create_new_group", + "create_overwrite_group", +] +``` + +- [ ] **Step 9: Rebuild and run the tests** + +Run: `uv sync --group zarrs --reinstall-package zarrs-bindings` +Run: `uv run --group zarrs pytest tests/zarrs/test_node.py -v` +Expected: 8 passed (4 tests × 2 store params). The MemoryStore param proves the full Rust→Python callback bridge; LocalStore proves the native path. + +- [ ] **Step 10: Commit** + +```bash +git add zarrs-bindings/src src/zarr/zarrs tests/zarrs/test_node.py +git commit -m "feat: zarrs store bridge and group creation" +``` + +--- + +### Task 5: Array creation + read_metadata + +**Files:** +- Modify: `zarrs-bindings/src/node.rs` +- Modify: `zarrs-bindings/src/lib.rs` +- Modify: `src/zarr/zarrs/_api.py`, `src/zarr/zarrs/__init__.py` +- Test: `tests/zarrs/test_node.py` + +- [ ] **Step 1: Add failing tests to `tests/zarrs/test_node.py`** + +Extend the imports: + +```python +import json + +import numpy as np + +from tests.zarrs.conftest import array_metadata +from zarr.errors import NodeNotFoundError +from zarr.zarrs import create_new_array, create_overwrite_array, read_metadata +``` + +(If `from tests.zarrs.conftest import ...` fails at collection, use a relative import `from .conftest import array_metadata` — `tests` is a package.) + +Add tests: + +```python +async def test_create_new_array(store: Store) -> None: + await create_new_array(array_metadata(), store, "arr") + arr = zarr.open_array(store=store, path="arr", mode="r") + assert arr.shape == (8, 8) + assert arr.chunks == (4, 4) + assert arr.dtype == np.dtype("uint16") + + +async def test_create_new_array_existing_node(store: Store) -> None: + await create_new_array(array_metadata(), store, "arr") + with pytest.raises(NodeExistsError): + await create_new_array(array_metadata(), store, "arr") + + +async def test_create_overwrite_array(store: Store) -> None: + zarr.create_group(store=store, path="arr") + await create_overwrite_array(array_metadata(), store, "arr") + arr = zarr.open_array(store=store, path="arr", mode="r") + assert arr.shape == (8, 8) + + +async def test_read_metadata_matches_stored_document(store: Store) -> None: + await create_new_array(array_metadata(), store, "arr") + observed = await read_metadata(store, "arr") + raw = await store.get("arr/zarr.json", prototype=default_buffer_prototype()) + assert raw is not None + assert observed == json.loads(raw.to_bytes()) + + +async def test_read_metadata_zarr_python_group(store: Store) -> None: + zarr.create_group(store=store, path="g", attributes={"a": 1}) + observed = await read_metadata(store, "g") + assert observed["node_type"] == "group" + assert observed["attributes"] == {"a": 1} + + +async def test_read_metadata_missing(store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await read_metadata(store, "nope") +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `uv run --group zarrs pytest tests/zarrs/test_node.py -v` +Expected: FAIL with `ImportError: cannot import name 'create_new_array'` + +- [ ] **Step 3: Add Rust functions to `zarrs-bindings/src/node.rs`** + +Extend the `use` block: + +```rust +use zarrs::array::Array; +use zarrs::metadata::ArrayMetadata; +use zarrs::node::Node; + +use crate::NodeNotFoundError; +``` + +Append: + +```rust +#[pyfunction] +pub(crate) fn create_array( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + overwrite: bool, +) -> PyResult<()> { + let storage = resolve_store(store)?; + let metadata = ArrayMetadata::try_from(metadata_json.as_str()).map_err(value_err)?; + py.detach(move || { + let node_path = parse_node_path(&path)?; + prepare_target(&storage, &node_path, overwrite)?; + let array = Array::new_with_metadata(storage, &path, metadata).map_err(value_err)?; + array.store_metadata().map_err(runtime_err) + }) +} + +#[pyfunction] +pub(crate) fn read_metadata( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, +) -> PyResult { + let storage = resolve_store(store)?; + py.detach(move || { + let node = Node::open(&storage, &path) + .map_err(|e| NodeNotFoundError::new_err(e.to_string()))?; + serde_json::to_string(node.metadata()).map_err(runtime_err) + }) +} +``` + +Register both in `lib.rs`: + +```rust + m.add_function(wrap_pyfunction!(node::create_array, m)?)?; + m.add_function(wrap_pyfunction!(node::read_metadata, m)?)?; +``` + +- [ ] **Step 4: Add Python wrappers to `src/zarr/zarrs/_api.py`** + +```python +async def create_new_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create an array at `path` from a v2 or v3 array metadata document. + + Raises `NodeExistsError` if any node already exists at `path`. + """ + with _translate_errors(): + await asyncio.to_thread( + _zb.create_array, resolve_store(store), _node_path(path), json.dumps(metadata), False + ) + + +async def create_overwrite_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create an array at `path`, deleting any existing node (and its children) first.""" + with _translate_errors(): + await asyncio.to_thread( + _zb.create_array, resolve_store(store), _node_path(path), json.dumps(metadata), True + ) + + +async def read_metadata( + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> dict[str, JSON]: + """Read the metadata document of the array or group at `path`. + + Raises `zarr.errors.NodeNotFoundError` if no node exists there. + """ + with _translate_errors(): + raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) + result: dict[str, JSON] = json.loads(raw) + return result +``` + +Add `create_new_array`, `create_overwrite_array`, `read_metadata` to the `__init__.py` import and `__all__`. + +- [ ] **Step 5: Rebuild and test** + +Run: `cargo check --manifest-path zarrs-bindings/Cargo.toml` → success +Run: `uv sync --group zarrs --reinstall-package zarrs-bindings` +Run: `uv run --group zarrs pytest tests/zarrs/test_node.py -v` +Expected: all pass (20 = 10 tests × 2 stores). Note: `test_read_metadata_matches_stored_document` asserts zarrs round-trips the document zarrs itself wrote; if zarrs normalizes a field zarr-python emits differently (e.g. drops a `null` `dimension_names`), adjust the *fixture* (`array_metadata`) to drop the field, not the assertion. + +- [ ] **Step 6: Commit** + +```bash +git add zarrs-bindings/src src/zarr/zarrs tests/zarrs +git commit -m "feat: zarrs-backed array creation and metadata reads" +``` + +--- + +### Task 6: delete_node + list_children + +**Files:** +- Modify: `zarrs-bindings/src/node.rs`, `zarrs-bindings/src/lib.rs` +- Modify: `src/zarr/zarrs/_api.py`, `src/zarr/zarrs/__init__.py` +- Test: `tests/zarrs/test_node.py` + +- [ ] **Step 1: Add failing tests to `tests/zarrs/test_node.py`** + +```python +from zarr.zarrs import delete_node, list_children + + +async def test_delete_node(store: Store) -> None: + arr = zarr.create_array(store=store, name="doomed", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + await delete_node(store, "doomed") + assert not await store.exists("doomed/zarr.json") + assert not await store.exists("doomed/c/0") + + +async def test_delete_node_missing(store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await delete_node(store, "nope") + + +async def test_list_children(store: Store) -> None: + root = zarr.create_group(store=store) + root.create_group("sub_group", attributes={"kind": "group"}) + root.create_array("sub_array", shape=(4,), chunks=(2,), dtype="uint8") + children = await list_children(store, "") + by_path = dict(children) + assert set(by_path) == {"sub_group", "sub_array"} + assert by_path["sub_group"]["node_type"] == "group" + assert by_path["sub_array"]["node_type"] == "array" + + +async def test_list_children_missing(store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await list_children(store, "nope") +``` + +- [ ] **Step 2: Run to verify failure** — `uv run --group zarrs pytest tests/zarrs/test_node.py -v` → ImportError. + +- [ ] **Step 3: Add Rust functions to `node.rs`** + +```rust +#[pyfunction] +pub(crate) fn delete_node( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, +) -> PyResult<()> { + let storage = resolve_store(store)?; + py.detach(move || { + let node_path = parse_node_path(&path)?; + if !node_exists(&storage, &node_path).map_err(runtime_err)? { + return Err(NodeNotFoundError::new_err(format!( + "no node found at path {}", + node_path.as_str() + ))); + } + let prefix: StorePrefix = (&node_path).try_into().map_err(value_err)?; + storage.erase_prefix(&prefix).map_err(runtime_err) + }) +} + +#[pyfunction] +pub(crate) fn list_children( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, +) -> PyResult> { + let storage = resolve_store(store)?; + py.detach(move || { + let group = Group::open(storage, &path) + .map_err(|e| NodeNotFoundError::new_err(e.to_string()))?; + let children = group.children(false).map_err(runtime_err)?; + children + .into_iter() + .map(|node| { + let metadata = serde_json::to_string(node.metadata()).map_err(runtime_err)?; + Ok((node.path().as_str().to_string(), metadata)) + }) + .collect() + }) +} +``` + +Register both in `lib.rs` as before. + +- [ ] **Step 4: Add Python wrappers to `_api.py`** + +```python +async def delete_node( + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Delete the node at `path`, including all keys and child nodes under it. + + Raises `zarr.errors.NodeNotFoundError` if no node exists there. Deleting the + root node (`path=""`) clears the entire store. + """ + with _translate_errors(): + await asyncio.to_thread(_zb.delete_node, resolve_store(store), _node_path(path)) + + +async def list_children( + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> list[tuple[str, dict[str, JSON]]]: + """List the direct children of the group at `path` as + `(path, metadata_document)` pairs. Paths are store-relative (no leading `/`). + + Raises `zarr.errors.NodeNotFoundError` if no group exists at `path`. + """ + with _translate_errors(): + raw = await asyncio.to_thread(_zb.list_children, resolve_store(store), _node_path(path)) + return [(child_path.lstrip("/"), json.loads(doc)) for child_path, doc in raw] +``` + +Export both from `__init__.py`. + +- [ ] **Step 5: Rebuild and test** + +Run: `uv sync --group zarrs --reinstall-package zarrs-bindings && uv run --group zarrs pytest tests/zarrs/test_node.py -v` +Expected: all pass. + +- [ ] **Step 6: Commit** + +```bash +git add zarrs-bindings/src src/zarr/zarrs tests/zarrs +git commit -m "feat: zarrs-backed node deletion and child listing" +``` + +--- + +### Task 7: Whole-chunk I/O (decode/encode/raw/erase) + +**Files:** +- Create: `zarrs-bindings/src/chunk.rs` +- Modify: `zarrs-bindings/src/lib.rs` +- Modify: `src/zarr/zarrs/_api.py`, `src/zarr/zarrs/__init__.py` +- Test: `tests/zarrs/test_chunk.py` + +- [ ] **Step 1: Write the failing tests** — `tests/zarrs/test_chunk.py` + +```python +from __future__ import annotations + +import copy +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +import zarr +from tests.zarrs.conftest import array_metadata +from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec +from zarr.core.buffer.core import default_buffer_prototype +from zarr.zarrs import ( + create_new_array, + decode_chunk, + encode_chunk, + erase_chunk, + read_encoded_chunk, +) + +if TYPE_CHECKING: + from zarr.abc.store import Store + + +def _filled( + store: Store, **kwargs: Any +) -> tuple[np.ndarray[Any, np.dtype[Any]], dict[str, Any]]: + """Create an 8x8 array named 'a' via zarr-python, fill it with a ramp, and + return (data, metadata_document).""" + params: dict[str, Any] = {"shape": (8, 8), "chunks": (4, 4), "dtype": "uint16"} | kwargs + arr = zarr.create_array(store=store, name="a", **params) + data = np.arange(64, dtype=params["dtype"]).reshape(8, 8) + arr[:, :] = data + doc = dict(arr.metadata.to_dict()) + if params.get("zarr_format") == 2: + # v2 attributes live in .zattrs, not in the .zarray document + doc.pop("attributes", None) + return data, doc + + +@pytest.mark.parametrize("dtype", ["uint8", "int32", "float64"]) +async def test_decode_chunk_differential(store: Store, dtype: str) -> None: + data, meta = _filled(store, dtype=dtype) + observed = await decode_chunk(meta, store, "a", (1, 0)) + np.testing.assert_array_equal(observed, data[4:8, 0:4]) + + +@pytest.mark.parametrize( + "compressors", [None, (GzipCodec(),), (ZstdCodec(),), (BloscCodec(cname="lz4"),)] +) +async def test_decode_chunk_codecs(store: Store, compressors: Any) -> None: + data, meta = _filled(store, compressors=compressors) + observed = await decode_chunk(meta, store, "a", (0, 1)) + np.testing.assert_array_equal(observed, data[0:4, 4:8]) + + +async def test_decode_chunk_v2(store: Store) -> None: + data, meta = _filled(store, zarr_format=2) + observed = await decode_chunk(meta, store, "a", (1, 1)) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_decode_chunk_sharding(store: Store) -> None: + # with sharding, the metadata chunk grid is the shard grid + data, meta = _filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await decode_chunk(meta, store, "a", (1, 1)) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_decode_chunk_missing_returns_fill_value(store: Store) -> None: + arr = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 + ) + meta = dict(arr.metadata.to_dict()) + observed = await decode_chunk(meta, store, "a", (0, 0)) + np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) + + +async def test_decode_chunk_selection_not_implemented(store: Store) -> None: + _, meta = _filled(store) + with pytest.raises(NotImplementedError): + await decode_chunk(meta, store, "a", (0, 0), selection=(slice(0, 2), slice(0, 2))) + + +async def test_decode_chunk_metadata_view(store: Store) -> None: + # the read-only-view case: decode with a metadata document the store never saw + data, meta = _filled(store, dtype="uint16", compressors=None) + view = copy.deepcopy(meta) + view["data_type"] = "uint8" + view["shape"] = [8, 16] + view["chunk_grid"]["configuration"]["chunk_shape"] = [4, 8] + observed = await decode_chunk(view, store, "a", (1, 0)) + np.testing.assert_array_equal(observed, data[4:8, 0:4].view("uint8")) + + +async def test_encode_chunk_differential(store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a") + value = np.arange(16, dtype="uint16").reshape(4, 4) + await encode_chunk(meta, store, "a", (0, 1), value) + arr = zarr.open_array(store=store, path="a", mode="r") + np.testing.assert_array_equal(arr[0:4, 4:8], value) + + +async def test_encode_chunk_shape_mismatch(store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a") + with pytest.raises(ValueError, match="chunk shape"): + await encode_chunk(meta, store, "a", (0, 0), np.zeros((2, 2), dtype="uint16")) + + +async def test_read_encoded_chunk_matches_store(store: Store) -> None: + _, meta = _filled(store) + raw = await read_encoded_chunk(meta, store, "a", (0, 0)) + expected = await store.get("a/c/0/0", prototype=default_buffer_prototype()) + assert expected is not None + assert raw == expected.to_bytes() + + +async def test_read_encoded_chunk_missing_returns_none(store: Store) -> None: + arr = zarr.create_array(store=store, name="empty", shape=(8, 8), chunks=(4, 4), dtype="uint16") + meta = dict(arr.metadata.to_dict()) + assert await read_encoded_chunk(meta, store, "empty", (0, 0)) is None + + +async def test_erase_chunk(store: Store) -> None: + data, meta = _filled(store) + assert await store.exists("a/c/0/0") + await erase_chunk(meta, store, "a", (0, 0)) + assert not await store.exists("a/c/0/0") + arr = zarr.open_array(store=store, path="a", mode="r") + np.testing.assert_array_equal(arr[0:4, 0:4], np.zeros((4, 4), dtype="uint16")) +``` + +- [ ] **Step 2: Run to verify failure** — `uv run --group zarrs pytest tests/zarrs/test_chunk.py -v` → ImportError. + +- [ ] **Step 3: Create `zarrs-bindings/src/chunk.rs`** + +```rust +use pyo3::exceptions::PyNotImplementedError; +use pyo3::prelude::*; +use pyo3::types::PyBytes; +use zarrs::array::{Array, ArrayBytes}; +use zarrs::metadata::ArrayMetadata; +use zarrs::storage::ReadableWritableListableStorage; + +use crate::store::resolve_store; +use crate::{runtime_err, value_err}; + +type DynArray = Array; + +/// Construct an Array view from an explicit metadata document, without +/// consulting the store for metadata. +fn array_view( + storage: ReadableWritableListableStorage, + path: &str, + metadata_json: &str, +) -> PyResult { + let metadata = ArrayMetadata::try_from(metadata_json).map_err(value_err)?; + Array::new_with_metadata(storage, path, metadata).map_err(value_err) +} + +#[pyfunction] +pub(crate) fn retrieve_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, +) -> PyResult> { + let storage = resolve_store(store)?; + let data = py.detach(move || -> PyResult> { + let array = array_view(storage, &path, &metadata_json)?; + let bytes: ArrayBytes<'static> = + array.retrieve_chunk(&chunk_coords).map_err(runtime_err)?; + let fixed = bytes.into_fixed().map_err(|_| { + PyNotImplementedError::new_err("variable-length data types are not supported") + })?; + Ok(fixed.into_owned()) + })?; + Ok(PyBytes::new(py, &data).unbind()) +} + +#[pyfunction] +pub(crate) fn retrieve_encoded_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, +) -> PyResult>> { + let storage = resolve_store(store)?; + let data = py.detach(move || -> PyResult>> { + let array = array_view(storage, &path, &metadata_json)?; + array + .retrieve_encoded_chunk(&chunk_coords) + .map_err(runtime_err) + })?; + Ok(data.map(|d| PyBytes::new(py, &d).unbind())) +} + +#[pyfunction] +pub(crate) fn store_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, + data: Vec, +) -> PyResult<()> { + let storage = resolve_store(store)?; + py.detach(move || { + let array = array_view(storage, &path, &metadata_json)?; + array + .store_chunk(&chunk_coords, ArrayBytes::new_flen(data)) + .map_err(runtime_err) + }) +} + +#[pyfunction] +pub(crate) fn erase_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, +) -> PyResult<()> { + let storage = resolve_store(store)?; + py.detach(move || { + let array = array_view(storage, &path, &metadata_json)?; + array.erase_chunk(&chunk_coords).map_err(runtime_err) + }) +} +``` + +Register in `lib.rs`: add `mod chunk;` and + +```rust + m.add_function(wrap_pyfunction!(chunk::retrieve_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::retrieve_encoded_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::store_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::erase_chunk, m)?)?; +``` + +- [ ] **Step 4: Add Python wrappers to `_api.py`** + +Extend imports: + +```python +from typing import Any + +import numpy as np +import numpy.typing as npt +``` + +Add: + +```python +def _chunk_dtype_and_shape( + metadata: Mapping[str, JSON], +) -> tuple[np.dtype[Any], tuple[int, ...]]: + """Resolve the numpy dtype and chunk shape from a metadata document, using + zarr-python's own metadata parsing.""" + from zarr.core.metadata.v2 import ArrayV2Metadata + from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata + + if metadata.get("zarr_format") == 3: + meta3 = ArrayV3Metadata.from_dict(dict(metadata)) + grid = meta3.chunk_grid + if not isinstance(grid, RegularChunkGridMetadata): + raise NotImplementedError("only regular chunk grids are supported") + return meta3.data_type.to_native_dtype(), grid.chunk_shape + meta2 = ArrayV2Metadata.from_dict(dict(metadata)) + return meta2.dtype.to_native_dtype(), meta2.chunks + + +async def decode_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + selection: tuple[slice | int, ...] | None = None, + options: ZarrsOptions | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode the chunk at `chunk_coords` of the array described by + `metadata`, located at `path` in `store`. + + The metadata document is authoritative: it is not read from the store. + Missing chunks decode to the fill value. `selection` (a chunk-relative + subset) is not implemented yet. + """ + if selection is not None: + raise NotImplementedError("chunk subset selection is not implemented yet") + raw = await asyncio.to_thread( + _zb.retrieve_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + ) + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + return np.frombuffer(raw, dtype=dtype).reshape(chunk_shape) + + +async def read_encoded_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: ZarrsOptions | None = None, +) -> bytes | None: + """Read the raw, still-encoded bytes of the chunk at `chunk_coords`, or + `None` if the chunk does not exist. No codecs are applied.""" + result: bytes | None = await asyncio.to_thread( + _zb.retrieve_encoded_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + ) + return result + + +async def encode_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + value: npt.ArrayLike, + *, + options: ZarrsOptions | None = None, +) -> None: + """Encode `value` with the codecs in `metadata` and store it as the chunk + at `chunk_coords`. `value` must match the chunk shape exactly.""" + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + arr = np.ascontiguousarray(np.asarray(value, dtype=dtype)) + if arr.shape != chunk_shape: + raise ValueError(f"value shape {arr.shape} does not match chunk shape {chunk_shape}") + await asyncio.to_thread( + _zb.store_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + arr.tobytes(), + ) + + +async def erase_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: ZarrsOptions | None = None, +) -> None: + """Delete the chunk at `chunk_coords`. Deleting a missing chunk is a no-op.""" + await asyncio.to_thread( + _zb.erase_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + ) +``` + +Export `decode_chunk`, `read_encoded_chunk`, `encode_chunk`, `erase_chunk` from `__init__.py`. + +- [ ] **Step 5: Rebuild and test** + +Run: `cargo check --manifest-path zarrs-bindings/Cargo.toml` → success +Run: `uv sync --group zarrs --reinstall-package zarrs-bindings` +Run: `uv run --group zarrs pytest tests/zarrs/test_chunk.py -v` +Expected: all pass. Likely first-run issues and their fixes: + - v2 differential test fails on dtype byte order → constrain the v2 test to `dtype=" None` annotations, which the code above has). + +- [ ] **Step 3: Re-run the full zarrs suite** — `uv run --group zarrs pytest tests/zarrs -v` → all pass. + +- [ ] **Step 4: Verify the rest of the test suite is unaffected** + +Run: `uv run pytest tests/test_array.py tests/test_group.py -x -q` +Expected: pass (no production code outside `src/zarr/zarrs/` changed). + +- [ ] **Step 5: Commit** + +```bash +git add -A +git commit -m "chore: lint fixes and changelog for zarr.zarrs" +``` + +--- + +### Task 9: CI workflow + +**Files:** +- Create: `.github/workflows/zarrs.yml` + +- [ ] **Step 1: Create `.github/workflows/zarrs.yml`** (action SHAs copied from `.github/workflows/test.yml` — keep them identical so dependabot groups them) + +```yaml +name: Zarrs bindings + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + fetch-depth: 0 # hatch-vcs needs tags to compute zarr's version + persist-credentials: false + - name: Install uv + uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + with: + python-version: '3.12' + - name: Run zarrs bindings tests + # the ubuntu runner image ships a Rust toolchain; the maturin build + # backend is fetched by uv on demand + run: uv run --group zarrs pytest tests/zarrs -v +``` + +- [ ] **Step 2: Validate the workflow** + +Run: `uvx zizmor .github/workflows/zarrs.yml` +Expected: no findings (matches the repo's zizmor policy). + +- [ ] **Step 3: Commit** + +```bash +git add .github/workflows/zarrs.yml +git commit -m "ci: test job for zarrs bindings" +``` + +--- + +## Out of scope for this plan (later phases, per spec) + +- `decode_region` / `encode_region` and chunk-subset `selection` (Phase 2: zarrs `retrieve_array_subset` / `partial_decoder`). +- `ZarrsOptions` fields (concurrency, checksum validation, direct IO), obstore native path, benchmarks (Phase 3). +- Variable-length data types, non-regular chunk grids, fancy indexing. +- Publishing the `zarrs-bindings` wheel / a `zarr[zarrs]` extra on PyPI. From 2eb3b6fff2bb61ecca9885bc19b22d36b3e54136 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 08:26:10 +0200 Subject: [PATCH 03/75] feat: scaffold zarrs-bindings PyO3 crate Create zarrs-bindings/ with Cargo.toml, pyproject.toml (maturin build backend), and src/lib.rs (exceptions + version function). Wire into the root pyproject.toml via a new `zarrs` dependency group and [tool.uv.sources]. Add zarrs-bindings/target/ to .gitignore. Co-Authored-By: Claude Fable 5 --- .gitignore | 3 + pyproject.toml | 8 + uv.lock | 35 + zarrs-bindings/Cargo.lock | 1819 +++++++++++++++++++++++++++++++++ zarrs-bindings/Cargo.toml | 18 + zarrs-bindings/pyproject.toml | 14 + zarrs-bindings/src/lib.rs | 38 + 7 files changed, 1935 insertions(+) create mode 100644 zarrs-bindings/Cargo.lock create mode 100644 zarrs-bindings/Cargo.toml create mode 100644 zarrs-bindings/pyproject.toml create mode 100644 zarrs-bindings/src/lib.rs diff --git a/.gitignore b/.gitignore index 3284865d6c..e5474b7c1c 100644 --- a/.gitignore +++ b/.gitignore @@ -94,3 +94,6 @@ zarr.egg-info/ # zarr-metadata package lockfile (a library, not an app) packages/zarr-metadata/uv.lock + +# zarrs-bindings Rust build artifacts +zarrs-bindings/target/ diff --git a/pyproject.toml b/pyproject.toml index 9f6005f981..90368ed37a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,6 +7,7 @@ exclude = [ "/.github", "/bench", "/docs", + "/zarrs-bindings", ] [project] @@ -134,6 +135,10 @@ dev = [ "universal-pathlib", "mypy", ] +zarrs = [ + {include-group = "test"}, + "zarrs-bindings", +] [tool.coverage.report] exclude_also = [ @@ -491,3 +496,6 @@ ignore-words-list = "astroid" [project.entry-points.pytest11] zarr = "zarr.testing" + +[tool.uv.sources] +zarrs-bindings = { path = "zarrs-bindings" } diff --git a/uv.lock b/uv.lock index ab2ff8b1e7..2b188848a9 100644 --- a/uv.lock +++ b/uv.lock @@ -4057,6 +4057,21 @@ test = [ { name = "tomlkit" }, { name = "uv" }, ] +zarrs = [ + { name = "coverage" }, + { name = "hypothesis" }, + { name = "numpydoc" }, + { name = "pytest" }, + { name = "pytest-accept" }, + { name = "pytest-asyncio" }, + { name = "pytest-benchmark" }, + { name = "pytest-codspeed" }, + { name = "pytest-cov" }, + { name = "pytest-xdist" }, + { name = "tomlkit" }, + { name = "uv" }, + { name = "zarrs-bindings" }, +] [package.metadata] requires-dist = [ @@ -4162,3 +4177,23 @@ test = [ { name = "tomlkit" }, { name = "uv" }, ] +zarrs = [ + { name = "coverage", specifier = ">=7.10" }, + { name = "hypothesis" }, + { name = "numpydoc" }, + { name = "pytest" }, + { name = "pytest-accept" }, + { name = "pytest-asyncio" }, + { name = "pytest-benchmark" }, + { name = "pytest-codspeed" }, + { name = "pytest-cov" }, + { name = "pytest-xdist" }, + { name = "tomlkit" }, + { name = "uv" }, + { name = "zarrs-bindings", directory = "zarrs-bindings" }, +] + +[[package]] +name = "zarrs-bindings" +version = "0.1.0" +source = { directory = "zarrs-bindings" } diff --git a/zarrs-bindings/Cargo.lock b/zarrs-bindings/Cargo.lock new file mode 100644 index 0000000000..a86d4e26e2 --- /dev/null +++ b/zarrs-bindings/Cargo.lock @@ -0,0 +1,1819 @@ +# This file is automatically @generated by Cargo. +# It is not intended for manual editing. +version = 4 + +[[package]] +name = "adler2" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" + +[[package]] +name = "ahash" +version = "0.8.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" +dependencies = [ + "cfg-if", + "getrandom 0.3.4", + "once_cell", + "version_check", + "zerocopy", +] + +[[package]] +name = "aho-corasick" +version = "1.1.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" +dependencies = [ + "memchr", +] + +[[package]] +name = "allocator-api2" +version = "0.2.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" + +[[package]] +name = "anyhow" +version = "1.0.102" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" + +[[package]] +name = "async-lock" +version = "3.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "290f7f2596bd5b78a9fec8088ccd89180d7f9f55b94b0576823bbbdc72ee8311" +dependencies = [ + "event-listener", + "event-listener-strategy", + "pin-project-lite", +] + +[[package]] +name = "async-trait" +version = "0.1.89" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "auto_impl" +version = "1.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ffdcb70bdbc4d478427380519163274ac86e52916e10f0a8889adf0f96d3fee7" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "autocfg" +version = "1.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" + +[[package]] +name = "base64" +version = "0.22.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" + +[[package]] +name = "bitflags" +version = "2.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" + +[[package]] +name = "blosc-src" +version = "0.3.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a9046dd58971db0226346fde214143d16a6eb12f535b5320d0ea94fcea420631" +dependencies = [ + "cc", + "libz-sys", + "lz4-sys", + "snappy_src", + "zstd-sys", +] + +[[package]] +name = "blusc" +version = "0.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b4e0c17eaa785d2673fe58c22fc817946c2330ed47f3d9f79835d65950d32a45" +dependencies = [ + "flate2", + "lz4_flex", + "pkg-config", + "snap", + "zstd", +] + +[[package]] +name = "bumpalo" +version = "3.20.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" + +[[package]] +name = "bytemuck" +version = "1.25.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" +dependencies = [ + "bytemuck_derive", +] + +[[package]] +name = "bytemuck_derive" +version = "1.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "byteorder" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" + +[[package]] +name = "bytes" +version = "1.11.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" + +[[package]] +name = "cc" +version = "1.2.63" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f" +dependencies = [ + "find-msvc-tools", + "jobserver", + "libc", + "shlex", +] + +[[package]] +name = "cfg-if" +version = "1.0.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" + +[[package]] +name = "concurrent-queue" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "convert_case" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "633458d4ef8c78b72454de2d54fd6ab2e60f9e02be22f3c6104cdc8a4e0fceb9" +dependencies = [ + "unicode-segmentation", +] + +[[package]] +name = "crc32c" +version = "0.6.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" +dependencies = [ + "rustc_version", +] + +[[package]] +name = "crc32fast" +version = "1.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "crossbeam-channel" +version = "0.5.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-deque" +version = "0.8.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" +dependencies = [ + "crossbeam-epoch", + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-epoch" +version = "0.9.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" +dependencies = [ + "crossbeam-utils", +] + +[[package]] +name = "crossbeam-utils" +version = "0.8.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" + +[[package]] +name = "crunchy" +version = "0.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" + +[[package]] +name = "derive_more" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134" +dependencies = [ + "derive_more-impl", +] + +[[package]] +name = "derive_more-impl" +version = "2.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" +dependencies = [ + "convert_case", + "proc-macro2", + "quote", + "rustc_version", + "syn", + "unicode-xid", +] + +[[package]] +name = "either" +version = "1.16.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" + +[[package]] +name = "equivalent" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" + +[[package]] +name = "event-listener" +version = "5.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e13b66accf52311f30a0db42147dadea9850cb48cd070028831ae5f5d4b856ab" +dependencies = [ + "concurrent-queue", + "parking", + "pin-project-lite", +] + +[[package]] +name = "event-listener-strategy" +version = "0.5.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" +dependencies = [ + "event-listener", + "pin-project-lite", +] + +[[package]] +name = "find-msvc-tools" +version = "0.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" + +[[package]] +name = "flate2" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" +dependencies = [ + "crc32fast", + "miniz_oxide", +] + +[[package]] +name = "foldhash" +version = "0.1.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" + +[[package]] +name = "foldhash" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" + +[[package]] +name = "futures" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" +dependencies = [ + "futures-channel", + "futures-core", + "futures-executor", + "futures-io", + "futures-sink", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-channel" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" +dependencies = [ + "futures-core", + "futures-sink", +] + +[[package]] +name = "futures-core" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" + +[[package]] +name = "futures-executor" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" +dependencies = [ + "futures-core", + "futures-task", + "futures-util", +] + +[[package]] +name = "futures-io" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" + +[[package]] +name = "futures-macro" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "futures-sink" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" + +[[package]] +name = "futures-task" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" + +[[package]] +name = "futures-util" +version = "0.3.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" +dependencies = [ + "futures-channel", + "futures-core", + "futures-io", + "futures-macro", + "futures-sink", + "futures-task", + "memchr", + "pin-project-lite", + "slab", +] + +[[package]] +name = "getrandom" +version = "0.3.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" +dependencies = [ + "cfg-if", + "js-sys", + "libc", + "r-efi 5.3.0", + "wasip2", + "wasm-bindgen", +] + +[[package]] +name = "getrandom" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +dependencies = [ + "cfg-if", + "libc", + "r-efi 6.0.0", + "wasip2", + "wasip3", +] + +[[package]] +name = "half" +version = "2.7.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" +dependencies = [ + "bytemuck", + "cfg-if", + "crunchy", + "zerocopy", +] + +[[package]] +name = "hashbrown" +version = "0.15.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" +dependencies = [ + "foldhash 0.1.5", +] + +[[package]] +name = "hashbrown" +version = "0.16.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" +dependencies = [ + "allocator-api2", + "equivalent", + "foldhash 0.2.0", +] + +[[package]] +name = "hashbrown" +version = "0.17.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" + +[[package]] +name = "heck" +version = "0.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" + +[[package]] +name = "id-arena" +version = "2.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" + +[[package]] +name = "indexmap" +version = "2.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" +dependencies = [ + "equivalent", + "hashbrown 0.17.1", + "serde", + "serde_core", +] + +[[package]] +name = "inventory" +version = "0.3.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a4f0c30c76f2f4ccee3fe55a2435f691ca00c0e4bd87abe4f4a851b1d4dac39b" +dependencies = [ + "rustversion", +] + +[[package]] +name = "itertools" +version = "0.14.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" +dependencies = [ + "either", +] + +[[package]] +name = "itoa" +version = "1.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" + +[[package]] +name = "jobserver" +version = "0.1.34" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" +dependencies = [ + "getrandom 0.3.4", + "libc", +] + +[[package]] +name = "js-sys" +version = "0.3.100" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f2025f20d7a4fa7785846e7b63d10a76d3f1cee98ee5cb79ea59703f95e42162" +dependencies = [ + "cfg-if", + "futures-util", + "wasm-bindgen", +] + +[[package]] +name = "leb128fmt" +version = "0.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" + +[[package]] +name = "libc" +version = "0.2.186" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" + +[[package]] +name = "libz-sys" +version = "1.1.29" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "85bc9657773828b90eeb625adff10eeac83cc21bbfd8e23a03eaa8a33c9e28d9" +dependencies = [ + "cc", + "libc", + "pkg-config", + "vcpkg", +] + +[[package]] +name = "link-cplusplus" +version = "1.0.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f78c730aaa7d0b9336a299029ea49f9ee53b0ed06e9202e8cb7db9bae7b8c82" +dependencies = [ + "cc", +] + +[[package]] +name = "lock_api" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" +dependencies = [ + "scopeguard", +] + +[[package]] +name = "log" +version = "0.4.32" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" + +[[package]] +name = "lru" +version = "0.16.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f66e8d5d03f609abc3a39e6f08e4164ebf1447a732906d39eb9b99b7919ef39" +dependencies = [ + "hashbrown 0.16.1", +] + +[[package]] +name = "lz4-sys" +version = "1.11.1+lz4-1.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6bd8c0d6c6ed0cd30b3652886bb8711dc4bb01d637a68105a3d5158039b418e6" +dependencies = [ + "cc", + "libc", +] + +[[package]] +name = "lz4_flex" +version = "0.12.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "90071f8077f8e40adfc4b7fe9cd495ce316263f19e75c2211eeff3fdf475a3d9" +dependencies = [ + "twox-hash", +] + +[[package]] +name = "matrixmultiply" +version = "0.3.10" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a06de3016e9fae57a36fd14dba131fccf49f74b40b7fbdb472f96e361ec71a08" +dependencies = [ + "autocfg", + "rawpointer", +] + +[[package]] +name = "memchr" +version = "2.8.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" + +[[package]] +name = "miniz_oxide" +version = "0.8.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" +dependencies = [ + "adler2", + "simd-adler32", +] + +[[package]] +name = "moka" +version = "0.12.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046" +dependencies = [ + "crossbeam-channel", + "crossbeam-epoch", + "crossbeam-utils", + "equivalent", + "parking_lot", + "portable-atomic", + "smallvec", + "tagptr", + "uuid", +] + +[[package]] +name = "monostate" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb4cc965c89dd0615a9e822ff8002f7633d2466143d51bd58693e4b2c75aabad" +dependencies = [ + "monostate-impl", + "serde", + "serde_core", +] + +[[package]] +name = "monostate-impl" +version = "1.0.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "23f5b99488110875b5904839d396c2cdfaf241ff6622638acb879cc7effad5de" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "ndarray" +version = "0.17.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d" +dependencies = [ + "matrixmultiply", + "num-complex", + "num-integer", + "num-traits", + "portable-atomic", + "portable-atomic-util", + "rawpointer", +] + +[[package]] +name = "num" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" +dependencies = [ + "num-bigint", + "num-complex", + "num-integer", + "num-iter", + "num-rational", + "num-traits", +] + +[[package]] +name = "num-bigint" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" +dependencies = [ + "num-integer", + "num-traits", +] + +[[package]] +name = "num-complex" +version = "0.4.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" +dependencies = [ + "bytemuck", + "num-traits", +] + +[[package]] +name = "num-integer" +version = "0.1.46" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" +dependencies = [ + "num-traits", +] + +[[package]] +name = "num-iter" +version = "0.1.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" +dependencies = [ + "autocfg", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-rational" +version = "0.4.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" +dependencies = [ + "num-bigint", + "num-integer", + "num-traits", +] + +[[package]] +name = "num-traits" +version = "0.2.19" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" +dependencies = [ + "autocfg", +] + +[[package]] +name = "once_cell" +version = "1.21.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" + +[[package]] +name = "page_size" +version = "0.6.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "30d5b2194ed13191c1999ae0704b7839fb18384fa22e49b57eeaa97d79ce40da" +dependencies = [ + "libc", + "winapi", +] + +[[package]] +name = "parking" +version = "2.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" + +[[package]] +name = "parking_lot" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" +dependencies = [ + "lock_api", + "parking_lot_core", +] + +[[package]] +name = "parking_lot_core" +version = "0.9.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" +dependencies = [ + "cfg-if", + "libc", + "redox_syscall", + "smallvec", + "windows-link", +] + +[[package]] +name = "paste" +version = "1.0.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" + +[[package]] +name = "pathdiff" +version = "0.2.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df94ce210e5bc13cb6651479fa48d14f601d9858cfe0467f43ae157023b938d3" + +[[package]] +name = "pin-project-lite" +version = "0.2.17" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" + +[[package]] +name = "pkg-config" +version = "0.3.33" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" + +[[package]] +name = "portable-atomic" +version = "1.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" + +[[package]] +name = "portable-atomic-util" +version = "0.2.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" +dependencies = [ + "portable-atomic", +] + +[[package]] +name = "positioned-io" +version = "0.3.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d4ec4b80060f033312b99b6874025d9503d2af87aef2dd4c516e253fbfcdada7" +dependencies = [ + "byteorder", + "libc", + "winapi", +] + +[[package]] +name = "prettyplease" +version = "0.2.37" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" +dependencies = [ + "proc-macro2", + "syn", +] + +[[package]] +name = "proc-macro2" +version = "1.0.106" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "pyo3" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91fd8e38a3b50ed1167fb981cd6fd60147e091784c427b8f7183a7ee32c31c12" +dependencies = [ + "libc", + "once_cell", + "portable-atomic", + "pyo3-build-config", + "pyo3-ffi", + "pyo3-macros", +] + +[[package]] +name = "pyo3-build-config" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e368e7ddfdeb98c9bca7f8383be1648fd84ab466bf2bc015e94008db6d35611e" +dependencies = [ + "target-lexicon", +] + +[[package]] +name = "pyo3-ffi" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7f29e10af80b1f7ccaf7f69eace800a03ecd13e883acfacc1e5d0988605f651e" +dependencies = [ + "libc", + "pyo3-build-config", +] + +[[package]] +name = "pyo3-macros" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df6e520eff47c45997d2fc7dd8214b25dd1310918bbb2642156ef66a67f29813" +dependencies = [ + "proc-macro2", + "pyo3-macros-backend", + "quote", + "syn", +] + +[[package]] +name = "pyo3-macros-backend" +version = "0.28.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c4cdc218d835738f81c2338f822078af45b4afdf8b2e33cbb5916f108b813acb" +dependencies = [ + "heck", + "proc-macro2", + "pyo3-build-config", + "quote", + "syn", +] + +[[package]] +name = "quick_cache" +version = "0.6.23" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3a3db184a8b66cfe87f0263a1de147a6b554c864d1767c6f7fa4eb0e5497b565" +dependencies = [ + "ahash", + "equivalent", + "hashbrown 0.16.1", + "parking_lot", +] + +[[package]] +name = "quote" +version = "1.0.45" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +dependencies = [ + "proc-macro2", +] + +[[package]] +name = "r-efi" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" + +[[package]] +name = "r-efi" +version = "6.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" + +[[package]] +name = "rawpointer" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" + +[[package]] +name = "rayon" +version = "1.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" +dependencies = [ + "either", + "rayon-core", +] + +[[package]] +name = "rayon-core" +version = "1.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" +dependencies = [ + "crossbeam-deque", + "crossbeam-utils", +] + +[[package]] +name = "rayon_iter_concurrent_limit" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d09ee01023de07fa073ce14c37cbe0a9e099c6b0b60a29cf4af6d04d9553fed7" +dependencies = [ + "rayon", +] + +[[package]] +name = "redox_syscall" +version = "0.5.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" +dependencies = [ + "bitflags", +] + +[[package]] +name = "regex" +version = "1.12.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba" +dependencies = [ + "aho-corasick", + "memchr", + "regex-automata", + "regex-syntax", +] + +[[package]] +name = "regex-automata" +version = "0.4.14" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" +dependencies = [ + "aho-corasick", + "memchr", + "regex-syntax", +] + +[[package]] +name = "regex-syntax" +version = "0.8.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" + +[[package]] +name = "rustc_version" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" +dependencies = [ + "semver", +] + +[[package]] +name = "rustversion" +version = "1.0.22" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" + +[[package]] +name = "same-file" +version = "1.0.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" +dependencies = [ + "winapi-util", +] + +[[package]] +name = "scopeguard" +version = "1.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" + +[[package]] +name = "semver" +version = "1.0.28" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" +dependencies = [ + "serde_core", + "serde_derive", +] + +[[package]] +name = "serde_core" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" +dependencies = [ + "serde_derive", +] + +[[package]] +name = "serde_derive" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "serde_json" +version = "1.0.150" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" +dependencies = [ + "indexmap", + "itoa", + "memchr", + "serde", + "serde_core", + "zmij", +] + +[[package]] +name = "serde_repr" +version = "0.1.20" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "shlex" +version = "2.0.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" + +[[package]] +name = "simd-adler32" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" + +[[package]] +name = "slab" +version = "0.4.12" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" + +[[package]] +name = "smallvec" +version = "1.15.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" + +[[package]] +name = "snap" +version = "1.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" + +[[package]] +name = "snappy_src" +version = "0.2.5+snappy.1.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4e1432067a55bcfb1fd522d2aca6537a4fcea32bba87ea86921226d14f9bad53" +dependencies = [ + "cc", + "link-cplusplus", +] + +[[package]] +name = "syn" +version = "2.0.117" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + +[[package]] +name = "tagptr" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" + +[[package]] +name = "target-lexicon" +version = "0.13.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" + +[[package]] +name = "thiserror" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" +dependencies = [ + "thiserror-impl", +] + +[[package]] +name = "thiserror-impl" +version = "2.0.18" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "thread_local" +version = "1.1.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185" +dependencies = [ + "cfg-if", +] + +[[package]] +name = "tinyvec" +version = "1.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" +dependencies = [ + "tinyvec_macros", +] + +[[package]] +name = "tinyvec_macros" +version = "0.1.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" + +[[package]] +name = "twox-hash" +version = "2.1.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" + +[[package]] +name = "unicode-ident" +version = "1.0.24" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" + +[[package]] +name = "unicode-segmentation" +version = "1.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" + +[[package]] +name = "unicode-xid" +version = "0.2.6" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" + +[[package]] +name = "unsafe_cell_slice" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6659959f702dcdaad77bd6e42a9409a32ceccc06943ec93c8a4306be00eb6cf1" + +[[package]] +name = "uuid" +version = "1.23.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" +dependencies = [ + "getrandom 0.4.2", + "js-sys", + "wasm-bindgen", +] + +[[package]] +name = "vcpkg" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" + +[[package]] +name = "version_check" +version = "0.9.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" + +[[package]] +name = "walkdir" +version = "2.5.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" +dependencies = [ + "same-file", + "winapi-util", +] + +[[package]] +name = "wasip2" +version = "1.0.3+wasi-0.2.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" +dependencies = [ + "wit-bindgen 0.57.1", +] + +[[package]] +name = "wasip3" +version = "0.4.0+wasi-0.3.0-rc-2026-01-06" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" +dependencies = [ + "wit-bindgen 0.51.0", +] + +[[package]] +name = "wasm-bindgen" +version = "0.2.123" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a254a4b10c19a76f09a27640e7ffbf9bc30bf67e16a3bf28aaefa4920fe81563" +dependencies = [ + "cfg-if", + "once_cell", + "rustversion", + "wasm-bindgen-macro", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-macro" +version = "0.2.123" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "24a40fc75b0ec6f3746ceb10d36f53a93dcd68a93b11b6445983945d79eba0dc" +dependencies = [ + "quote", + "wasm-bindgen-macro-support", +] + +[[package]] +name = "wasm-bindgen-macro-support" +version = "0.2.123" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "908f34bd9b9ce3d4caf07b72dfab63d61504d156856c6bd3cd87fa350cf3985b" +dependencies = [ + "bumpalo", + "proc-macro2", + "quote", + "syn", + "wasm-bindgen-shared", +] + +[[package]] +name = "wasm-bindgen-shared" +version = "0.2.123" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7acbf7616c27b194bbb550bf77ed0c2c3e5b7fd1260a93082b95fb7f47959b92" +dependencies = [ + "unicode-ident", +] + +[[package]] +name = "wasm-encoder" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" +dependencies = [ + "leb128fmt", + "wasmparser", +] + +[[package]] +name = "wasm-metadata" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" +dependencies = [ + "anyhow", + "indexmap", + "wasm-encoder", + "wasmparser", +] + +[[package]] +name = "wasmparser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" +dependencies = [ + "bitflags", + "hashbrown 0.15.5", + "indexmap", + "semver", +] + +[[package]] +name = "winapi" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" +dependencies = [ + "winapi-i686-pc-windows-gnu", + "winapi-x86_64-pc-windows-gnu", +] + +[[package]] +name = "winapi-i686-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" + +[[package]] +name = "winapi-util" +version = "0.1.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" +dependencies = [ + "windows-sys", +] + +[[package]] +name = "winapi-x86_64-pc-windows-gnu" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" + +[[package]] +name = "windows-link" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" + +[[package]] +name = "windows-sys" +version = "0.61.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" +dependencies = [ + "windows-link", +] + +[[package]] +name = "wit-bindgen" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" +dependencies = [ + "wit-bindgen-rust-macro", +] + +[[package]] +name = "wit-bindgen" +version = "0.57.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" + +[[package]] +name = "wit-bindgen-core" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" +dependencies = [ + "anyhow", + "heck", + "wit-parser", +] + +[[package]] +name = "wit-bindgen-rust" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" +dependencies = [ + "anyhow", + "heck", + "indexmap", + "prettyplease", + "syn", + "wasm-metadata", + "wit-bindgen-core", + "wit-component", +] + +[[package]] +name = "wit-bindgen-rust-macro" +version = "0.51.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" +dependencies = [ + "anyhow", + "prettyplease", + "proc-macro2", + "quote", + "syn", + "wit-bindgen-core", + "wit-bindgen-rust", +] + +[[package]] +name = "wit-component" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" +dependencies = [ + "anyhow", + "bitflags", + "indexmap", + "log", + "serde", + "serde_derive", + "serde_json", + "wasm-encoder", + "wasm-metadata", + "wasmparser", + "wit-parser", +] + +[[package]] +name = "wit-parser" +version = "0.244.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" +dependencies = [ + "anyhow", + "id-arena", + "indexmap", + "log", + "semver", + "serde", + "serde_derive", + "serde_json", + "unicode-xid", + "wasmparser", +] + +[[package]] +name = "zarrs" +version = "0.23.13" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8132307b8fc041fd21f68c7987103fb6e038b11f9838c16ec43b798f5480ccf5" +dependencies = [ + "async-lock", + "base64", + "blosc-src", + "blusc", + "bytemuck", + "bytes", + "crc32c", + "derive_more", + "flate2", + "getrandom 0.3.4", + "half", + "inventory", + "itertools", + "itoa", + "libz-sys", + "log", + "lru", + "moka", + "ndarray", + "num", + "num-complex", + "paste", + "quick_cache", + "rayon", + "rayon_iter_concurrent_limit", + "serde", + "serde_json", + "thiserror", + "thread_local", + "unsafe_cell_slice", + "uuid", + "zarrs_chunk_grid", + "zarrs_chunk_key_encoding", + "zarrs_codec", + "zarrs_data_type", + "zarrs_filesystem", + "zarrs_metadata", + "zarrs_metadata_ext", + "zarrs_plugin", + "zarrs_storage", + "zstd", +] + +[[package]] +name = "zarrs-bindings" +version = "0.1.0" +dependencies = [ + "pyo3", + "serde_json", + "zarrs", +] + +[[package]] +name = "zarrs_chunk_grid" +version = "0.5.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1cf67386fd96a0336cd3e5ab5ca6cb14e0e05aee80f1acae8c4d3cf562a8bb65" +dependencies = [ + "derive_more", + "inventory", + "itertools", + "rayon", + "thiserror", + "tinyvec", + "zarrs_metadata", + "zarrs_plugin", +] + +[[package]] +name = "zarrs_chunk_key_encoding" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9040e7feaa92d1904d492acd0cd91b97214f1791c5b5738e6c05b2ca4145a382" +dependencies = [ + "derive_more", + "inventory", + "zarrs_metadata", + "zarrs_plugin", + "zarrs_storage", +] + +[[package]] +name = "zarrs_codec" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "383a129a6a0cbb2c80cdba23809e5cab85159756464b7d0f112468a495c128da" +dependencies = [ + "async-trait", + "bytemuck", + "derive_more", + "futures", + "inventory", + "itertools", + "rayon", + "thiserror", + "unsafe_cell_slice", + "zarrs_chunk_grid", + "zarrs_data_type", + "zarrs_metadata", + "zarrs_plugin", + "zarrs_storage", +] + +[[package]] +name = "zarrs_data_type" +version = "0.9.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1bc7c594c9363278fcd9db4c205514f009944206eb093ea7ad40b85f50009f31" +dependencies = [ + "derive_more", + "half", + "inventory", + "num", + "paste", + "serde", + "serde_json", + "thiserror", + "zarrs_metadata", + "zarrs_plugin", +] + +[[package]] +name = "zarrs_filesystem" +version = "0.3.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "270efeb0181651aee5460b3232f2fc83e91bd646cefe75001d1c8f9a4f3abf81" +dependencies = [ + "bytes", + "derive_more", + "itertools", + "libc", + "page_size", + "pathdiff", + "positioned-io", + "thiserror", + "walkdir", + "zarrs_storage", +] + +[[package]] +name = "zarrs_metadata" +version = "0.7.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d60c4c363a8a302d7babb3c29017850a7b4e0af6ca5f9ba2946263a185b62fea" +dependencies = [ + "derive_more", + "half", + "monostate", + "serde", + "serde_json", + "thiserror", +] + +[[package]] +name = "zarrs_metadata_ext" +version = "0.4.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2048e07848ca99c7450518e0584929300b1b6a3cf442f18b26ffd3520814bd5b" +dependencies = [ + "derive_more", + "monostate", + "num", + "serde", + "serde_json", + "serde_repr", + "thiserror", + "zarrs_metadata", +] + +[[package]] +name = "zarrs_plugin" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "5cbe0ed432aee86856f70ca33be36eaf4a0dae21ab730750d9280a7ca1e95046" +dependencies = [ + "paste", + "regex", + "serde_json", + "thiserror", +] + +[[package]] +name = "zarrs_storage" +version = "0.4.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7d098796d2ed4cf94896569615101e0432e870a7665396da5cc32300fb68f7c1" +dependencies = [ + "auto_impl", + "bytes", + "derive_more", + "itertools", + "thiserror", + "unsafe_cell_slice", +] + +[[package]] +name = "zerocopy" +version = "0.8.52" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f" +dependencies = [ + "zerocopy-derive", +] + +[[package]] +name = "zerocopy-derive" +version = "0.8.52" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" +dependencies = [ + "proc-macro2", + "quote", + "syn", +] + +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + +[[package]] +name = "zstd" +version = "0.13.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" +dependencies = [ + "zstd-safe", +] + +[[package]] +name = "zstd-safe" +version = "7.2.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" +dependencies = [ + "zstd-sys", +] + +[[package]] +name = "zstd-sys" +version = "2.0.16+zstd.1.5.7" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" +dependencies = [ + "cc", + "pkg-config", +] diff --git a/zarrs-bindings/Cargo.toml b/zarrs-bindings/Cargo.toml new file mode 100644 index 0000000000..be97fa4b5a --- /dev/null +++ b/zarrs-bindings/Cargo.toml @@ -0,0 +1,18 @@ +[package] +name = "zarrs-bindings" +version = "0.1.0" +edition = "2024" +rust-version = "1.91" +publish = false + +[lib] +name = "_zarrs_bindings" +crate-type = ["cdylib"] + +[dependencies] +pyo3 = { version = "0.28", features = ["abi3-py312"] } +serde_json = "1" +zarrs = "0.23" + +[profile.release] +lto = "thin" diff --git a/zarrs-bindings/pyproject.toml b/zarrs-bindings/pyproject.toml new file mode 100644 index 0000000000..66eff31f36 --- /dev/null +++ b/zarrs-bindings/pyproject.toml @@ -0,0 +1,14 @@ +[build-system] +requires = ["maturin>=1.7,<2"] +build-backend = "maturin" + +[project] +name = "zarrs-bindings" +version = "0.1.0" +description = "PyO3 bindings to the zarrs Rust crate, consumed by zarr.zarrs" +requires-python = ">=3.12" +license = "MIT" + +[tool.maturin] +module-name = "_zarrs_bindings" +strip = true diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs new file mode 100644 index 0000000000..803104a63c --- /dev/null +++ b/zarrs-bindings/src/lib.rs @@ -0,0 +1,38 @@ +use pyo3::exceptions::{PyRuntimeError, PyValueError}; +use pyo3::prelude::*; + +pyo3::create_exception!( + _zarrs_bindings, + NodeExistsError, + PyValueError, + "A node already exists at the given path." +); +pyo3::create_exception!( + _zarrs_bindings, + NodeNotFoundError, + PyValueError, + "No node was found at the given path." +); + +#[allow(dead_code)] +pub(crate) fn runtime_err(err: impl std::fmt::Display) -> PyErr { + PyRuntimeError::new_err(err.to_string()) +} + +#[allow(dead_code)] +pub(crate) fn value_err(err: impl std::fmt::Display) -> PyErr { + PyValueError::new_err(err.to_string()) +} + +#[pyfunction] +fn version() -> &'static str { + env!("CARGO_PKG_VERSION") +} + +#[pymodule] +fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { + m.add("NodeExistsError", m.py().get_type::())?; + m.add("NodeNotFoundError", m.py().get_type::())?; + m.add_function(wrap_pyfunction!(version, m)?)?; + Ok(()) +} From bfc0b645ad7645fb02616fc9e8414475c62339da Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 09:07:26 +0200 Subject: [PATCH 04/75] fix: single-source zarrs-bindings version from Cargo.toml Co-Authored-By: Claude Fable 5 --- uv.lock | 1 - zarrs-bindings/Cargo.toml | 2 ++ zarrs-bindings/pyproject.toml | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/uv.lock b/uv.lock index 2b188848a9..a1c94a2d4b 100644 --- a/uv.lock +++ b/uv.lock @@ -4195,5 +4195,4 @@ zarrs = [ [[package]] name = "zarrs-bindings" -version = "0.1.0" source = { directory = "zarrs-bindings" } diff --git a/zarrs-bindings/Cargo.toml b/zarrs-bindings/Cargo.toml index be97fa4b5a..e0f381f416 100644 --- a/zarrs-bindings/Cargo.toml +++ b/zarrs-bindings/Cargo.toml @@ -4,6 +4,8 @@ version = "0.1.0" edition = "2024" rust-version = "1.91" publish = false +license = "MIT" +description = "PyO3 bindings to the zarrs Rust crate, consumed by zarr.zarrs" [lib] name = "_zarrs_bindings" diff --git a/zarrs-bindings/pyproject.toml b/zarrs-bindings/pyproject.toml index 66eff31f36..4212a64b56 100644 --- a/zarrs-bindings/pyproject.toml +++ b/zarrs-bindings/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "maturin" [project] name = "zarrs-bindings" -version = "0.1.0" +dynamic = ["version"] description = "PyO3 bindings to the zarrs Rust crate, consumed by zarr.zarrs" requires-python = ">=3.12" license = "MIT" From 5982c04d7658e194e46cfa204c8f0b9e07154c32 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 09:11:23 +0200 Subject: [PATCH 05/75] feat: add zarr.zarrs package skeleton and test scaffolding Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/__init__.py | 23 +++++++++++++++++++++ tests/zarrs/__init__.py | 0 tests/zarrs/conftest.py | 41 ++++++++++++++++++++++++++++++++++++++ tests/zarrs/test_api.py | 7 +++++++ 4 files changed, 71 insertions(+) create mode 100644 src/zarr/zarrs/__init__.py create mode 100644 tests/zarrs/__init__.py create mode 100644 tests/zarrs/conftest.py create mode 100644 tests/zarrs/test_api.py diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py new file mode 100644 index 0000000000..3aa871bf32 --- /dev/null +++ b/src/zarr/zarrs/__init__.py @@ -0,0 +1,23 @@ +""" +Low-level functional API for zarr hierarchies, backed by the Rust +[`zarrs`](https://zarrs.dev) crate. + +This subpackage is experimental. It requires the `zarrs-bindings` package +(in-repo Rust crate; install for development with `uv sync --group zarrs`). + +All array routines take an explicit metadata document (a `dict` matching the +`zarr.json` / `.zarray` document) rather than reading metadata from the store, +which makes read-only and virtual views possible. +""" + +try: + import _zarrs_bindings +except ImportError as e: + raise ImportError( + "zarr.zarrs requires the `zarrs-bindings` package, which is not installed. " + "It is built from the zarr-python repository: run `uv sync --group zarrs`." + ) from e + +__version__: str = _zarrs_bindings.version() + +__all__ = ["__version__"] diff --git a/tests/zarrs/__init__.py b/tests/zarrs/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/zarrs/conftest.py b/tests/zarrs/conftest.py new file mode 100644 index 0000000000..f54758d59f --- /dev/null +++ b/tests/zarrs/conftest.py @@ -0,0 +1,41 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import pytest + +pytest.importorskip("_zarrs_bindings", reason="zarrs-bindings is not installed") + +import zarr +from zarr.storage import LocalStore, MemoryStore + +if TYPE_CHECKING: + from pathlib import Path + + from zarr.abc.store import Store + + +@pytest.fixture(params=["memory", "local"]) +async def store(request: pytest.FixtureRequest, tmp_path: Path) -> Store: + """A writable store: MemoryStore exercises the generic Python-callback bridge, + LocalStore exercises the native zarrs filesystem store.""" + if request.param == "memory": + return await MemoryStore.open() + return await LocalStore.open(root=tmp_path / "store") + + +def array_metadata(**kwargs: Any) -> dict[str, Any]: + """Build an array metadata document using zarr-python itself, so the + documents fed to zarrs always match what zarr-python would write.""" + params: dict[str, Any] = { + "shape": (8, 8), + "chunks": (4, 4), + "dtype": "uint16", + "zarr_format": 3, + } | kwargs + arr = zarr.create_array(store=MemoryStore(), **params) + doc = dict(arr.metadata.to_dict()) + if params["zarr_format"] == 2: + # v2 attributes live in .zattrs, not in the .zarray document + doc.pop("attributes", None) + return doc diff --git a/tests/zarrs/test_api.py b/tests/zarrs/test_api.py new file mode 100644 index 0000000000..da1a9ecda8 --- /dev/null +++ b/tests/zarrs/test_api.py @@ -0,0 +1,7 @@ +from __future__ import annotations + + +def test_import() -> None: + import zarr.zarrs + + assert isinstance(zarr.zarrs.__version__, str) From e821d70915e2116fd54166770eaab18670bb8a5e Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 09:19:00 +0200 Subject: [PATCH 06/75] fix: pass exc_type to importorskip in zarrs conftest Co-Authored-By: Claude Fable 5 --- tests/zarrs/conftest.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/zarrs/conftest.py b/tests/zarrs/conftest.py index f54758d59f..b9b3bcbab1 100644 --- a/tests/zarrs/conftest.py +++ b/tests/zarrs/conftest.py @@ -4,7 +4,9 @@ import pytest -pytest.importorskip("_zarrs_bindings", reason="zarrs-bindings is not installed") +pytest.importorskip( + "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError +) import zarr from zarr.storage import LocalStore, MemoryStore From 88c0c4d1424d6b73396553d678bc7d3a087e9ae8 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 09:47:54 +0200 Subject: [PATCH 07/75] feat: sync store bridge for zarrs bindings Add StoreShim, a synchronous adapter over zarr's async Store ABC, and resolve_store, which maps a Store to either a native config dict (for LocalStore) or a StoreShim for Rust to call back into. Also convert the store fixture in tests/zarrs/conftest.py to an async generator with teardown so stores are properly closed after each test. Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_bridge.py | 100 +++++++++++++++++++++++++++++++++++++ tests/zarrs/conftest.py | 13 +++-- tests/zarrs/test_bridge.py | 45 +++++++++++++++++ 3 files changed, 155 insertions(+), 3 deletions(-) create mode 100644 src/zarr/zarrs/_bridge.py create mode 100644 tests/zarrs/test_bridge.py diff --git a/src/zarr/zarrs/_bridge.py b/src/zarr/zarrs/_bridge.py new file mode 100644 index 0000000000..d68c5df76f --- /dev/null +++ b/src/zarr/zarrs/_bridge.py @@ -0,0 +1,100 @@ +from __future__ import annotations + +import builtins +from typing import TYPE_CHECKING + +from zarr.abc.store import OffsetByteRequest, RangeByteRequest, SuffixByteRequest +from zarr.core.buffer.core import default_buffer_prototype +from zarr.core.sync import _collect_aiterator, sync +from zarr.storage import LocalStore + +if TYPE_CHECKING: + from zarr.abc.store import Store + +# Alias to avoid shadowing the `list` builtin with the `StoreShim.list` method +# in mypy's class-scope name resolution. +_list = builtins.list + + +class StoreShim: + """ + Synchronous adapter over an async `Store`, called from Rust worker threads. + + Each method blocks the calling thread by submitting a coroutine to the zarr + event-loop thread (`zarr.core.sync`). Methods must never be called from the + zarr event-loop thread itself; the Rust bindings only call them from + `asyncio.to_thread` worker threads. + """ + + def __init__(self, store: Store) -> None: + self._store = store + self._prototype = default_buffer_prototype() + + def get(self, key: str) -> bytes | None: + buf = sync(self._store.get(key, prototype=self._prototype)) + return None if buf is None else buf.to_bytes() + + def get_range(self, key: str, offset: int, length: int | None) -> bytes | None: + byte_range = ( + RangeByteRequest(offset, offset + length) + if length is not None + else OffsetByteRequest(offset) + ) + buf = sync(self._store.get(key, prototype=self._prototype, byte_range=byte_range)) + return None if buf is None else buf.to_bytes() + + def get_suffix(self, key: str, suffix: int) -> bytes | None: + buf = sync( + self._store.get(key, prototype=self._prototype, byte_range=SuffixByteRequest(suffix)) + ) + return None if buf is None else buf.to_bytes() + + def set(self, key: str, value: bytes) -> None: + sync(self._store.set(key, self._prototype.buffer.from_bytes(value))) + + def delete(self, key: str) -> None: + sync(self._store.delete(key)) + + def delete_prefix(self, prefix: str) -> None: + sync(self._store.delete_dir(prefix.rstrip("/"))) + + def getsize(self, key: str) -> int | None: + try: + return sync(self._store.getsize(key)) + except FileNotFoundError: + return None + + def getsize_prefix(self, prefix: str) -> int: + return sync(self._store.getsize_prefix(prefix.rstrip("/"))) + + def list(self) -> _list[str]: + return sorted(sync(_collect_aiterator(self._store.list()))) + + def list_prefix(self, prefix: str) -> _list[str]: + return sorted(sync(_collect_aiterator(self._store.list_prefix(prefix)))) + + def list_dir(self, prefix: str) -> tuple[_list[str], _list[str]]: + """Return `(keys, prefixes)` directly under `prefix`, as zarrs expects: + full keys, and child prefixes ending in `/`.""" + stripped = prefix.rstrip("/") + children = sorted(sync(_collect_aiterator(self._store.list_dir(stripped)))) + keys: _list[str] = [] + prefixes: _list[str] = [] + for child in children: + full = f"{stripped}/{child}" if stripped else child + if sync(self._store.exists(full)): + keys.append(full) + else: + prefixes.append(full + "/") + return keys, prefixes + + +def resolve_store(store: Store) -> StoreShim | dict[str, str]: + """ + Convert a zarr `Store` into the representation `_zarrs_bindings` expects: + a config dict for stores with a native Rust implementation, otherwise a + `StoreShim` that Rust calls back into. + """ + if isinstance(store, LocalStore) and not store.read_only: + return {"filesystem": str(store.root)} + return StoreShim(store) diff --git a/tests/zarrs/conftest.py b/tests/zarrs/conftest.py index b9b3bcbab1..b65cca447b 100644 --- a/tests/zarrs/conftest.py +++ b/tests/zarrs/conftest.py @@ -12,18 +12,25 @@ from zarr.storage import LocalStore, MemoryStore if TYPE_CHECKING: + from collections.abc import AsyncGenerator from pathlib import Path from zarr.abc.store import Store @pytest.fixture(params=["memory", "local"]) -async def store(request: pytest.FixtureRequest, tmp_path: Path) -> Store: +async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncGenerator[Store, None]: """A writable store: MemoryStore exercises the generic Python-callback bridge, LocalStore exercises the native zarrs filesystem store.""" + s: Store if request.param == "memory": - return await MemoryStore.open() - return await LocalStore.open(root=tmp_path / "store") + s = await MemoryStore.open() + else: + s = await LocalStore.open(root=tmp_path / "store") + try: + yield s + finally: + s.close() def array_metadata(**kwargs: Any) -> dict[str, Any]: diff --git a/tests/zarrs/test_bridge.py b/tests/zarrs/test_bridge.py new file mode 100644 index 0000000000..12fe01c7df --- /dev/null +++ b/tests/zarrs/test_bridge.py @@ -0,0 +1,45 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr.storage import LocalStore, MemoryStore +from zarr.zarrs._bridge import StoreShim, resolve_store + +if TYPE_CHECKING: + from pathlib import Path + + +def test_shim_get_set_delete() -> None: + shim = StoreShim(MemoryStore()) + assert shim.get("a/b") is None + shim.set("a/b", b"xyz") + assert shim.get("a/b") == b"xyz" + assert shim.get_range("a/b", 1, 1) == b"y" + assert shim.get_range("a/b", 1, None) == b"yz" + assert shim.get_suffix("a/b", 2) == b"yz" + assert shim.getsize("a/b") == 3 + assert shim.getsize("missing") is None + shim.delete("a/b") + assert shim.get("a/b") is None + + +def test_shim_listing() -> None: + shim = StoreShim(MemoryStore()) + shim.set("zarr.json", b"{}") + shim.set("a/zarr.json", b"{}") + shim.set("a/c/0/0", b"\x00") + assert shim.list() == ["a/c/0/0", "a/zarr.json", "zarr.json"] + assert shim.list_prefix("a/") == ["a/c/0/0", "a/zarr.json"] + assert shim.list_dir("a/") == (["a/zarr.json"], ["a/c/"]) + assert shim.list_dir("") == (["zarr.json"], ["a/"]) + assert shim.getsize_prefix("a/") == 3 + shim.delete_prefix("a/") + assert shim.list() == ["zarr.json"] + + +def test_resolve_store(tmp_path: Path) -> None: + local = LocalStore(tmp_path) + assert resolve_store(local) == {"filesystem": str(tmp_path)} + # read-only LocalStore must go through the shim so writes are rejected in Python + assert isinstance(resolve_store(LocalStore(tmp_path, read_only=True)), StoreShim) + assert isinstance(resolve_store(MemoryStore()), StoreShim) From 737e82e39bc593332d87bbe0e557f895624a045c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 10:03:30 +0200 Subject: [PATCH 08/75] refactor: review polish for zarrs store bridge Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_bridge.py | 5 +++++ tests/zarrs/conftest.py | 4 ++-- tests/zarrs/test_bridge.py | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/zarr/zarrs/_bridge.py b/src/zarr/zarrs/_bridge.py index d68c5df76f..e7632647ad 100644 --- a/src/zarr/zarrs/_bridge.py +++ b/src/zarr/zarrs/_bridge.py @@ -80,6 +80,11 @@ def list_dir(self, prefix: str) -> tuple[_list[str], _list[str]]: children = sorted(sync(_collect_aiterator(self._store.list_dir(stripped)))) keys: _list[str] = [] prefixes: _list[str] = [] + # A child is classified as a key iff it exists as one. Zarr hierarchies + # never store a bare key alongside same-named subkeys (e.g. "a" and + # "a/b"), so a name is never both a key and a prefix. + # TODO: replace the per-child exists() round-trip with a single listing + # pass when this becomes a bottleneck (remote stores). for child in children: full = f"{stripped}/{child}" if stripped else child if sync(self._store.exists(full)): diff --git a/tests/zarrs/conftest.py b/tests/zarrs/conftest.py index b65cca447b..678065e5e1 100644 --- a/tests/zarrs/conftest.py +++ b/tests/zarrs/conftest.py @@ -12,14 +12,14 @@ from zarr.storage import LocalStore, MemoryStore if TYPE_CHECKING: - from collections.abc import AsyncGenerator + from collections.abc import AsyncIterator from pathlib import Path from zarr.abc.store import Store @pytest.fixture(params=["memory", "local"]) -async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncGenerator[Store, None]: +async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncIterator[Store]: """A writable store: MemoryStore exercises the generic Python-callback bridge, LocalStore exercises the native zarrs filesystem store.""" s: Store diff --git a/tests/zarrs/test_bridge.py b/tests/zarrs/test_bridge.py index 12fe01c7df..2b88c047dd 100644 --- a/tests/zarrs/test_bridge.py +++ b/tests/zarrs/test_bridge.py @@ -19,6 +19,8 @@ def test_shim_get_set_delete() -> None: assert shim.get_suffix("a/b", 2) == b"yz" assert shim.getsize("a/b") == 3 assert shim.getsize("missing") is None + assert shim.get_range("missing", 0, 1) is None + assert shim.get_suffix("missing", 1) is None shim.delete("a/b") assert shim.get("a/b") is None From 6b8f60388fd4ea5ae1f372b09eec57cd1633d5aa Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 10:22:46 +0200 Subject: [PATCH 09/75] feat: zarrs store bridge and group creation Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/__init__.py | 15 ++- src/zarr/zarrs/_api.py | 75 +++++++++++++ tests/zarrs/test_node.py | 48 ++++++++ zarrs-bindings/src/lib.rs | 6 +- zarrs-bindings/src/node.rs | 50 +++++++++ zarrs-bindings/src/store.rs | 216 ++++++++++++++++++++++++++++++++++++ 6 files changed, 407 insertions(+), 3 deletions(-) create mode 100644 src/zarr/zarrs/_api.py create mode 100644 tests/zarrs/test_node.py create mode 100644 zarrs-bindings/src/node.rs create mode 100644 zarrs-bindings/src/store.rs diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py index 3aa871bf32..7e7af7644e 100644 --- a/src/zarr/zarrs/__init__.py +++ b/src/zarr/zarrs/__init__.py @@ -20,4 +20,17 @@ __version__: str = _zarrs_bindings.version() -__all__ = ["__version__"] +from zarr.zarrs._api import ( + NodeExistsError, + ZarrsOptions, + create_new_group, + create_overwrite_group, +) + +__all__ = [ + "NodeExistsError", + "ZarrsOptions", + "__version__", + "create_new_group", + "create_overwrite_group", +] diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py new file mode 100644 index 0000000000..cfe3f6ec2c --- /dev/null +++ b/src/zarr/zarrs/_api.py @@ -0,0 +1,75 @@ +from __future__ import annotations + +import asyncio +import json +from contextlib import contextmanager +from dataclasses import dataclass +from typing import TYPE_CHECKING + +import _zarrs_bindings as _zb + +from zarr.errors import NodeNotFoundError +from zarr.zarrs._bridge import resolve_store + +if TYPE_CHECKING: + from collections.abc import Iterator, Mapping + + from zarr.abc.store import Store + from zarr.core.common import JSON + +NodeExistsError = _zb.NodeExistsError +"""Raised by `create_new_*` when a node already exists at the target path.""" + + +@dataclass(frozen=True, slots=True) +class ZarrsOptions: + """Options for zarrs-backed operations. + + Currently empty: fields (concurrency limits, checksum validation) arrive in + a later phase. Accepting it now keeps signatures stable. + """ + + +def _node_path(path: str) -> str: + """Convert a zarr-python node path (`""`, `"foo/bar"`) to a zarrs node path + (`"/"`, `"/foo/bar"`).""" + return f"/{path.strip('/')}" + + +@contextmanager +def _translate_errors() -> Iterator[None]: + try: + yield + except _zb.NodeNotFoundError as err: + raise NodeNotFoundError(str(err)) from err + + +async def create_new_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create a group at `path` from a group metadata document. + + Raises `NodeExistsError` if any node already exists at `path`. + """ + with _translate_errors(): + await asyncio.to_thread( + _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), False + ) + + +async def create_overwrite_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create a group at `path`, deleting any existing node (and its children) first.""" + with _translate_errors(): + await asyncio.to_thread( + _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), True + ) diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py new file mode 100644 index 0000000000..19e9654ded --- /dev/null +++ b/tests/zarrs/test_node.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import pytest + +import zarr +from zarr.core.buffer.core import default_buffer_prototype +from zarr.zarrs import NodeExistsError, create_new_group, create_overwrite_group + +if TYPE_CHECKING: + from zarr.abc.store import Store + +GROUP_META: dict[str, Any] = { + "zarr_format": 3, + "node_type": "group", + "attributes": {"answer": 42}, +} + + +async def test_create_new_group(store: Store) -> None: + await create_new_group(GROUP_META, store, "foo") + group = zarr.open_group(store=store, path="foo", mode="r") + assert dict(group.attrs) == {"answer": 42} + + +async def test_create_new_group_at_root(store: Store) -> None: + await create_new_group(GROUP_META, store, "") + group = zarr.open_group(store=store, mode="r") + assert dict(group.attrs) == {"answer": 42} + + +async def test_create_new_group_existing_node(store: Store) -> None: + await create_new_group(GROUP_META, store, "foo") + with pytest.raises(NodeExistsError): + await create_new_group(GROUP_META, store, "foo") + + +async def test_create_overwrite_group(store: Store) -> None: + # an array and its chunks previously occupied the path; overwrite removes both + arr = zarr.create_array(store=store, name="foo", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + assert await store.exists("foo/c/0") + await create_overwrite_group(GROUP_META, store, "foo") + group = zarr.open_group(store=store, path="foo", mode="r") + assert dict(group.attrs) == {"answer": 42} + assert not await store.exists("foo/c/0") + assert await store.get("foo/zarr.json", prototype=default_buffer_prototype()) is not None diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs index 803104a63c..e83a71e568 100644 --- a/zarrs-bindings/src/lib.rs +++ b/zarrs-bindings/src/lib.rs @@ -1,6 +1,9 @@ use pyo3::exceptions::{PyRuntimeError, PyValueError}; use pyo3::prelude::*; +mod node; +mod store; + pyo3::create_exception!( _zarrs_bindings, NodeExistsError, @@ -14,12 +17,10 @@ pyo3::create_exception!( "No node was found at the given path." ); -#[allow(dead_code)] pub(crate) fn runtime_err(err: impl std::fmt::Display) -> PyErr { PyRuntimeError::new_err(err.to_string()) } -#[allow(dead_code)] pub(crate) fn value_err(err: impl std::fmt::Display) -> PyErr { PyValueError::new_err(err.to_string()) } @@ -34,5 +35,6 @@ fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add("NodeExistsError", m.py().get_type::())?; m.add("NodeNotFoundError", m.py().get_type::())?; m.add_function(wrap_pyfunction!(version, m)?)?; + m.add_function(wrap_pyfunction!(node::create_group, m)?)?; Ok(()) } diff --git a/zarrs-bindings/src/node.rs b/zarrs-bindings/src/node.rs new file mode 100644 index 0000000000..c557c66409 --- /dev/null +++ b/zarrs-bindings/src/node.rs @@ -0,0 +1,50 @@ +use pyo3::prelude::*; +use zarrs::group::Group; +use zarrs::metadata::GroupMetadata; +use zarrs::node::{node_exists, NodePath}; +use zarrs::storage::{ReadableWritableListableStorage, StorePrefix}; + +use crate::store::resolve_store; +use crate::{runtime_err, value_err, NodeExistsError}; + +pub(crate) fn parse_node_path(path: &str) -> PyResult { + NodePath::new(path).map_err(value_err) +} + +/// When a node exists at `node_path`: erase it (and everything under it) if +/// `overwrite`, otherwise raise `NodeExistsError`. +pub(crate) fn prepare_target( + storage: &ReadableWritableListableStorage, + node_path: &NodePath, + overwrite: bool, +) -> PyResult<()> { + if node_exists(storage, node_path).map_err(runtime_err)? { + if !overwrite { + return Err(NodeExistsError::new_err(format!( + "a node already exists at path {}", + node_path.as_str() + ))); + } + let prefix: StorePrefix = node_path.try_into().map_err(value_err)?; + storage.erase_prefix(&prefix).map_err(runtime_err)?; + } + Ok(()) +} + +#[pyfunction] +pub(crate) fn create_group( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + overwrite: bool, +) -> PyResult<()> { + let storage = resolve_store(store)?; + let metadata = GroupMetadata::try_from(metadata_json.as_str()).map_err(value_err)?; + py.detach(move || { + let node_path = parse_node_path(&path)?; + prepare_target(&storage, &node_path, overwrite)?; + let group = Group::new_with_metadata(storage, &path, metadata).map_err(value_err)?; + group.store_metadata().map_err(runtime_err) + }) +} diff --git a/zarrs-bindings/src/store.rs b/zarrs-bindings/src/store.rs new file mode 100644 index 0000000000..3ab8312061 --- /dev/null +++ b/zarrs-bindings/src/store.rs @@ -0,0 +1,216 @@ +use std::sync::Arc; + +use pyo3::exceptions::PyValueError; +use pyo3::prelude::*; +use pyo3::types::{PyBytes, PyDict}; +use zarrs::filesystem::FilesystemStore; +use zarrs::storage::byte_range::{ByteRange, ByteRangeIterator}; +use zarrs::storage::{ + Bytes, ListableStorageTraits, MaybeBytes, MaybeBytesIterator, OffsetBytesIterator, + ReadableStorageTraits, ReadableWritableListableStorage, StorageError, StoreKey, StoreKeys, + StoreKeysPrefixes, StorePrefix, WritableStorageTraits, +}; + +/// A zarrs store backed by a Python `zarr.zarrs._bridge.StoreShim`. +/// +/// Every method attaches to the Python interpreter and calls the shim, which +/// blocks on the zarr event loop. Blocking waits in Python release the GIL, so +/// the loop thread can make progress while a Rust worker waits here. +pub(crate) struct PyStore(Py); + +fn py_err(err: PyErr) -> StorageError { + StorageError::Other(err.to_string()) +} + +fn invalid(err: impl std::fmt::Display) -> StorageError { + StorageError::Other(err.to_string()) +} + +impl PyStore { + fn get_with_range( + &self, + key: &StoreKey, + range: Option<&ByteRange>, + ) -> Result { + Python::attach(|py| { + let shim = self.0.bind(py); + let result = match range { + None => shim.call_method1("get", (key.as_str(),)), + Some(ByteRange::FromStart(offset, length)) => { + shim.call_method1("get_range", (key.as_str(), *offset, *length)) + } + Some(ByteRange::Suffix(suffix)) => { + shim.call_method1("get_suffix", (key.as_str(), *suffix)) + } + } + .map_err(py_err)?; + if result.is_none() { + Ok(None) + } else { + let bytes: Vec = result.extract().map_err(py_err)?; + Ok(Some(Bytes::from(bytes))) + } + }) + } +} + +impl ReadableStorageTraits for PyStore { + fn get(&self, key: &StoreKey) -> Result { + self.get_with_range(key, None) + } + + fn get_partial_many<'a>( + &'a self, + key: &StoreKey, + byte_ranges: ByteRangeIterator<'a>, + ) -> Result, StorageError> { + let mut out = Vec::new(); + for byte_range in byte_ranges { + match self.get_with_range(key, Some(&byte_range))? { + Some(bytes) => out.push(Ok(bytes)), + None => return Ok(None), + } + } + Ok(Some(Box::new(out.into_iter()))) + } + + fn size_key(&self, key: &StoreKey) -> Result, StorageError> { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("getsize", (key.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err) + }) + } + + fn supports_get_partial(&self) -> bool { + true + } +} + +impl WritableStorageTraits for PyStore { + fn set(&self, key: &StoreKey, value: Bytes) -> Result<(), StorageError> { + Python::attach(|py| { + let data = PyBytes::new(py, &value); + self.0 + .bind(py) + .call_method1("set", (key.as_str(), data)) + .map_err(py_err)?; + Ok(()) + }) + } + + fn set_partial_many( + &self, + key: &StoreKey, + offset_values: OffsetBytesIterator, + ) -> Result<(), StorageError> { + // read-modify-write fallback provided by zarrs + zarrs::storage::store_set_partial_many(self, key, offset_values) + } + + fn supports_set_partial(&self) -> bool { + false + } + + fn erase(&self, key: &StoreKey) -> Result<(), StorageError> { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("delete", (key.as_str(),)) + .map_err(py_err)?; + Ok(()) + }) + } + + fn erase_prefix(&self, prefix: &StorePrefix) -> Result<(), StorageError> { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("delete_prefix", (prefix.as_str(),)) + .map_err(py_err)?; + Ok(()) + }) + } +} + +impl ListableStorageTraits for PyStore { + fn list(&self) -> Result { + Python::attach(|py| { + let keys: Vec = self + .0 + .bind(py) + .call_method0("list") + .map_err(py_err)? + .extract() + .map_err(py_err)?; + keys.into_iter() + .map(|k| StoreKey::new(k).map_err(invalid)) + .collect() + }) + } + + fn list_prefix(&self, prefix: &StorePrefix) -> Result { + Python::attach(|py| { + let keys: Vec = self + .0 + .bind(py) + .call_method1("list_prefix", (prefix.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err)?; + keys.into_iter() + .map(|k| StoreKey::new(k).map_err(invalid)) + .collect() + }) + } + + fn list_dir(&self, prefix: &StorePrefix) -> Result { + Python::attach(|py| { + let (keys, prefixes): (Vec, Vec) = self + .0 + .bind(py) + .call_method1("list_dir", (prefix.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err)?; + let keys = keys + .into_iter() + .map(|k| StoreKey::new(k).map_err(invalid)) + .collect::, StorageError>>()?; + let prefixes = prefixes + .into_iter() + .map(|p| StorePrefix::new(p).map_err(invalid)) + .collect::, StorageError>>()?; + Ok(StoreKeysPrefixes::new(keys, prefixes)) + }) + } + + fn size_prefix(&self, prefix: &StorePrefix) -> Result { + Python::attach(|py| { + self.0 + .bind(py) + .call_method1("getsize_prefix", (prefix.as_str(),)) + .map_err(py_err)? + .extract() + .map_err(py_err) + }) + } +} + +/// Convert the Python-side store representation (`zarr.zarrs._bridge.resolve_store` +/// output) into a zarrs storage handle. +pub(crate) fn resolve_store(obj: &Bound<'_, PyAny>) -> PyResult { + if let Ok(config) = obj.cast::() { + if let Some(root) = config.get_item("filesystem")? { + let root: String = root.extract()?; + let store = + FilesystemStore::new(root).map_err(|e| PyValueError::new_err(e.to_string()))?; + return Ok(Arc::new(store)); + } + return Err(PyValueError::new_err("unrecognized store configuration")); + } + Ok(Arc::new(PyStore(obj.clone().unbind()))) +} From 32573fd56f46bf6e06911a8958ce473901edd314 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 11:24:49 +0200 Subject: [PATCH 10/75] docs: document phase-1 error fidelity and creation race; cargo fmt Co-Authored-By: Claude Fable 5 --- .../specs/2026-06-11-zarrs-functional-api-design.md | 12 ++++++++---- src/zarr/zarrs/_api.py | 8 +++++++- zarrs-bindings/src/node.rs | 4 ++-- 3 files changed, 17 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md index ad28c987f8..ebf2854611 100644 --- a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md +++ b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md @@ -140,10 +140,14 @@ side releases the GIL during I/O and compute (reacquiring it only inside The binding layer raises a small set of typed exceptions defined in one place: `NodeExistsError`, `NodeNotFoundError`, and `ValueError` subclasses for -metadata-parse and decode failures. `zarr.zarrs` translates to zarr-python -native exception types where an obvious equivalent exists (e.g. -`zarr.errors.ContainsArrayError`). Store-callback exceptions from Python -propagate through Rust unchanged. +metadata-parse failures. In Phase 1 the translation surface is deliberately +small: `zarr.zarrs` re-raises the bindings' `NodeNotFoundError` as +`zarr.errors.NodeNotFoundError`; `NodeExistsError` is exposed as +`zarr.zarrs.NodeExistsError`. Exceptions raised by Python store callbacks are +flattened to a `RuntimeError` carrying the original message — the original +exception type and traceback are lost crossing the Rust boundary. Faithful +propagation of store-callback exceptions (and richer mapping onto +`zarr.errors` types) is deferred to a later phase. ## Testing diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index cfe3f6ec2c..a125cc2688 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -54,6 +54,8 @@ async def create_new_group( """Create a group at `path` from a group metadata document. Raises `NodeExistsError` if any node already exists at `path`. + Creation is not atomic with respect to concurrent writers: a concurrent + creation at the same path can race the existence check. """ with _translate_errors(): await asyncio.to_thread( @@ -68,7 +70,11 @@ async def create_overwrite_group( *, options: ZarrsOptions | None = None, ) -> None: - """Create a group at `path`, deleting any existing node (and its children) first.""" + """Create a group at `path`, deleting any existing node (and its children) first. + + Creation is not atomic with respect to concurrent writers: a concurrent + creation at the same path can race the existence check. + """ with _translate_errors(): await asyncio.to_thread( _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), True diff --git a/zarrs-bindings/src/node.rs b/zarrs-bindings/src/node.rs index c557c66409..3daa6f3666 100644 --- a/zarrs-bindings/src/node.rs +++ b/zarrs-bindings/src/node.rs @@ -1,11 +1,11 @@ use pyo3::prelude::*; use zarrs::group::Group; use zarrs::metadata::GroupMetadata; -use zarrs::node::{node_exists, NodePath}; +use zarrs::node::{NodePath, node_exists}; use zarrs::storage::{ReadableWritableListableStorage, StorePrefix}; use crate::store::resolve_store; -use crate::{runtime_err, value_err, NodeExistsError}; +use crate::{NodeExistsError, runtime_err, value_err}; pub(crate) fn parse_node_path(path: &str) -> PyResult { NodePath::new(path).map_err(value_err) From f6a95e428b9a146344988bbd057592a55156ed63 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 11:42:09 +0200 Subject: [PATCH 11/75] feat: zarrs-backed array creation and metadata reads Adds `create_array` and `read_metadata` pyfunctions to the zarrs-bindings crate, and exposes them as `create_new_array`, `create_overwrite_array`, and `read_metadata` in the `zarr.zarrs` subpackage. Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/__init__.py | 6 +++++ src/zarr/zarrs/_api.py | 52 ++++++++++++++++++++++++++++++++++++ tests/zarrs/test_node.py | 54 +++++++++++++++++++++++++++++++++++++- zarrs-bindings/src/lib.rs | 2 ++ zarrs-bindings/src/node.rs | 39 ++++++++++++++++++++++++--- 5 files changed, 149 insertions(+), 4 deletions(-) diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py index 7e7af7644e..2e8fd4b107 100644 --- a/src/zarr/zarrs/__init__.py +++ b/src/zarr/zarrs/__init__.py @@ -23,14 +23,20 @@ from zarr.zarrs._api import ( NodeExistsError, ZarrsOptions, + create_new_array, create_new_group, + create_overwrite_array, create_overwrite_group, + read_metadata, ) __all__ = [ "NodeExistsError", "ZarrsOptions", "__version__", + "create_new_array", "create_new_group", + "create_overwrite_array", "create_overwrite_group", + "read_metadata", ] diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index a125cc2688..c4851641c8 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -79,3 +79,55 @@ async def create_overwrite_group( await asyncio.to_thread( _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), True ) + + +async def create_new_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create an array at `path` from a v2 or v3 array metadata document. + + Raises `NodeExistsError` if any node already exists at `path`. Creation is + not atomic with respect to concurrent writers: a concurrent creation at the + same path can race the existence check. + """ + with _translate_errors(): + await asyncio.to_thread( + _zb.create_array, resolve_store(store), _node_path(path), json.dumps(metadata), False + ) + + +async def create_overwrite_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Create an array at `path`, deleting any existing node (and its children) + first. The delete-then-create sequence is not atomic with respect to + concurrent writers. + """ + with _translate_errors(): + await asyncio.to_thread( + _zb.create_array, resolve_store(store), _node_path(path), json.dumps(metadata), True + ) + + +async def read_metadata( + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> dict[str, JSON]: + """Read the metadata document of the array or group at `path`. + + Raises `zarr.errors.NodeNotFoundError` if no node exists there. + """ + with _translate_errors(): + raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) + result: dict[str, JSON] = json.loads(raw) + return result diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py index 19e9654ded..7601e7c2ea 100644 --- a/tests/zarrs/test_node.py +++ b/tests/zarrs/test_node.py @@ -1,12 +1,23 @@ from __future__ import annotations +import json from typing import TYPE_CHECKING, Any +import numpy as np import pytest import zarr +from tests.zarrs.conftest import array_metadata from zarr.core.buffer.core import default_buffer_prototype -from zarr.zarrs import NodeExistsError, create_new_group, create_overwrite_group +from zarr.errors import NodeNotFoundError +from zarr.zarrs import ( + NodeExistsError, + create_new_array, + create_new_group, + create_overwrite_array, + create_overwrite_group, + read_metadata, +) if TYPE_CHECKING: from zarr.abc.store import Store @@ -46,3 +57,44 @@ async def test_create_overwrite_group(store: Store) -> None: assert dict(group.attrs) == {"answer": 42} assert not await store.exists("foo/c/0") assert await store.get("foo/zarr.json", prototype=default_buffer_prototype()) is not None + + +async def test_create_new_array(store: Store) -> None: + await create_new_array(array_metadata(), store, "arr") + arr = zarr.open_array(store=store, path="arr", mode="r") + assert arr.shape == (8, 8) + assert arr.chunks == (4, 4) + assert arr.dtype == np.dtype("uint16") + + +async def test_create_new_array_existing_node(store: Store) -> None: + await create_new_array(array_metadata(), store, "arr") + with pytest.raises(NodeExistsError): + await create_new_array(array_metadata(), store, "arr") + + +async def test_create_overwrite_array(store: Store) -> None: + zarr.create_group(store=store, path="arr") + await create_overwrite_array(array_metadata(), store, "arr") + arr = zarr.open_array(store=store, path="arr", mode="r") + assert arr.shape == (8, 8) + + +async def test_read_metadata_matches_stored_document(store: Store) -> None: + await create_new_array(array_metadata(), store, "arr") + observed = await read_metadata(store, "arr") + raw = await store.get("arr/zarr.json", prototype=default_buffer_prototype()) + assert raw is not None + assert observed == json.loads(raw.to_bytes()) + + +async def test_read_metadata_zarr_python_group(store: Store) -> None: + zarr.create_group(store=store, path="g", attributes={"a": 1}) + observed = await read_metadata(store, "g") + assert observed["node_type"] == "group" + assert observed["attributes"] == {"a": 1} + + +async def test_read_metadata_missing(store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await read_metadata(store, "nope") diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs index e83a71e568..3c03578a61 100644 --- a/zarrs-bindings/src/lib.rs +++ b/zarrs-bindings/src/lib.rs @@ -35,6 +35,8 @@ fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add("NodeExistsError", m.py().get_type::())?; m.add("NodeNotFoundError", m.py().get_type::())?; m.add_function(wrap_pyfunction!(version, m)?)?; + m.add_function(wrap_pyfunction!(node::create_array, m)?)?; m.add_function(wrap_pyfunction!(node::create_group, m)?)?; + m.add_function(wrap_pyfunction!(node::read_metadata, m)?)?; Ok(()) } diff --git a/zarrs-bindings/src/node.rs b/zarrs-bindings/src/node.rs index 3daa6f3666..95642d45b8 100644 --- a/zarrs-bindings/src/node.rs +++ b/zarrs-bindings/src/node.rs @@ -1,11 +1,12 @@ use pyo3::prelude::*; +use zarrs::array::Array; use zarrs::group::Group; -use zarrs::metadata::GroupMetadata; -use zarrs::node::{NodePath, node_exists}; +use zarrs::metadata::{ArrayMetadata, GroupMetadata}; +use zarrs::node::{Node, NodePath, node_exists}; use zarrs::storage::{ReadableWritableListableStorage, StorePrefix}; use crate::store::resolve_store; -use crate::{NodeExistsError, runtime_err, value_err}; +use crate::{NodeExistsError, NodeNotFoundError, runtime_err, value_err}; pub(crate) fn parse_node_path(path: &str) -> PyResult { NodePath::new(path).map_err(value_err) @@ -48,3 +49,35 @@ pub(crate) fn create_group( group.store_metadata().map_err(runtime_err) }) } + +#[pyfunction] +pub(crate) fn create_array( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + overwrite: bool, +) -> PyResult<()> { + let storage = resolve_store(store)?; + let metadata = ArrayMetadata::try_from(metadata_json.as_str()).map_err(value_err)?; + py.detach(move || { + let node_path = parse_node_path(&path)?; + prepare_target(&storage, &node_path, overwrite)?; + let array = Array::new_with_metadata(storage, &path, metadata).map_err(value_err)?; + array.store_metadata().map_err(runtime_err) + }) +} + +#[pyfunction] +pub(crate) fn read_metadata( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, +) -> PyResult { + let storage = resolve_store(store)?; + py.detach(move || { + let node = + Node::open(&storage, &path).map_err(|e| NodeNotFoundError::new_err(e.to_string()))?; + serde_json::to_string(node.metadata()).map_err(runtime_err) + }) +} From 59fd58d09a891ad29b506fdca7cee9c81cd5f201 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 11:51:34 +0200 Subject: [PATCH 12/75] test: v2 array creation via zarrs; review polish Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_api.py | 5 ++--- tests/zarrs/test_node.py | 7 +++++++ zarrs-bindings/src/node.rs | 2 ++ 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index c4851641c8..598a435653 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -4,7 +4,7 @@ import json from contextlib import contextmanager from dataclasses import dataclass -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, cast import _zarrs_bindings as _zb @@ -129,5 +129,4 @@ async def read_metadata( """ with _translate_errors(): raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) - result: dict[str, JSON] = json.loads(raw) - return result + return cast("dict[str, JSON]", json.loads(raw)) diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py index 7601e7c2ea..6a178d2947 100644 --- a/tests/zarrs/test_node.py +++ b/tests/zarrs/test_node.py @@ -80,6 +80,13 @@ async def test_create_overwrite_array(store: Store) -> None: assert arr.shape == (8, 8) +async def test_create_new_array_v2(store: Store) -> None: + await create_new_array(array_metadata(zarr_format=2), store, "arr") + arr = zarr.open_array(store=store, path="arr", mode="r") + assert arr.metadata.zarr_format == 2 + assert arr.shape == (8, 8) + + async def test_read_metadata_matches_stored_document(store: Store) -> None: await create_new_array(array_metadata(), store, "arr") observed = await read_metadata(store, "arr") diff --git a/zarrs-bindings/src/node.rs b/zarrs-bindings/src/node.rs index 95642d45b8..c7833246e6 100644 --- a/zarrs-bindings/src/node.rs +++ b/zarrs-bindings/src/node.rs @@ -8,6 +8,8 @@ use zarrs::storage::{ReadableWritableListableStorage, StorePrefix}; use crate::store::resolve_store; use crate::{NodeExistsError, NodeNotFoundError, runtime_err, value_err}; +/// `path` arguments throughout this module are zarrs node paths, e.g. "/" or +/// "/foo/bar" (already normalized by the Python layer's `_node_path`). pub(crate) fn parse_node_path(path: &str) -> PyResult { NodePath::new(path).map_err(value_err) } From 2faace584d3708ecf4a596d371e76ba095074778 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 11:57:16 +0200 Subject: [PATCH 13/75] feat: zarrs-backed node deletion and child listing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add `delete_node` and `list_children` to the zarrs-bindings Rust crate and the `zarr.zarrs` Python subpackage. `delete_node` erases the node prefix via `erase_prefix`, raising `NodeNotFoundError` when the node is absent. `list_children` opens the target as a `Group` and returns direct children as `(path, metadata_document)` pairs. Both are covered by 4 new tests (× 2 stores = 8 parametrized cases); total suite: 34 passed. Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/__init__.py | 4 ++++ src/zarr/zarrs/_api.py | 37 +++++++++++++++++++++++++++++++++++++ tests/zarrs/test_node.py | 31 +++++++++++++++++++++++++++++++ zarrs-bindings/src/lib.rs | 2 ++ zarrs-bindings/src/node.rs | 37 +++++++++++++++++++++++++++++++++++++ 5 files changed, 111 insertions(+) diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py index 2e8fd4b107..7d4968cd40 100644 --- a/src/zarr/zarrs/__init__.py +++ b/src/zarr/zarrs/__init__.py @@ -27,6 +27,8 @@ create_new_group, create_overwrite_array, create_overwrite_group, + delete_node, + list_children, read_metadata, ) @@ -38,5 +40,7 @@ "create_new_group", "create_overwrite_array", "create_overwrite_group", + "delete_node", + "list_children", "read_metadata", ] diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index 598a435653..09c58d6585 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -130,3 +130,40 @@ async def read_metadata( with _translate_errors(): raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) return cast("dict[str, JSON]", json.loads(raw)) + + +async def delete_node( + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> None: + """Delete the node at `path`, including all keys and child nodes under it. + + Raises `zarr.errors.NodeNotFoundError` if no node exists there. Deleting + the root node (`path=""`) clears the entire store. + """ + with _translate_errors(): + await asyncio.to_thread(_zb.delete_node, resolve_store(store), _node_path(path)) + + +async def list_children( + store: Store, + path: str, + *, + options: ZarrsOptions | None = None, +) -> list[tuple[str, dict[str, JSON]]]: + """List the direct children of the group at `path` as + `(path, metadata_document)` pairs. Paths are store-relative (no leading + `/`). + + Raises `zarr.errors.NodeNotFoundError` if no group exists at `path`. + """ + with _translate_errors(): + raw: list[tuple[str, str]] = await asyncio.to_thread( + _zb.list_children, resolve_store(store), _node_path(path) + ) + return [ + (child_path.lstrip("/"), cast("dict[str, JSON]", json.loads(doc))) + for child_path, doc in raw + ] diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py index 6a178d2947..c5d46f472f 100644 --- a/tests/zarrs/test_node.py +++ b/tests/zarrs/test_node.py @@ -16,6 +16,8 @@ create_new_group, create_overwrite_array, create_overwrite_group, + delete_node, + list_children, read_metadata, ) @@ -105,3 +107,32 @@ async def test_read_metadata_zarr_python_group(store: Store) -> None: async def test_read_metadata_missing(store: Store) -> None: with pytest.raises(NodeNotFoundError): await read_metadata(store, "nope") + + +async def test_delete_node(store: Store) -> None: + arr = zarr.create_array(store=store, name="doomed", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + await delete_node(store, "doomed") + assert not await store.exists("doomed/zarr.json") + assert not await store.exists("doomed/c/0") + + +async def test_delete_node_missing(store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await delete_node(store, "nope") + + +async def test_list_children(store: Store) -> None: + root = zarr.create_group(store=store) + root.create_group("sub_group", attributes={"kind": "group"}) + root.create_array("sub_array", shape=(4,), chunks=(2,), dtype="uint8") + children = await list_children(store, "") + by_path = dict(children) + assert set(by_path) == {"sub_group", "sub_array"} + assert by_path["sub_group"]["node_type"] == "group" + assert by_path["sub_array"]["node_type"] == "array" + + +async def test_list_children_missing(store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await list_children(store, "nope") diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs index 3c03578a61..6a98d4ec53 100644 --- a/zarrs-bindings/src/lib.rs +++ b/zarrs-bindings/src/lib.rs @@ -37,6 +37,8 @@ fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(version, m)?)?; m.add_function(wrap_pyfunction!(node::create_array, m)?)?; m.add_function(wrap_pyfunction!(node::create_group, m)?)?; + m.add_function(wrap_pyfunction!(node::delete_node, m)?)?; + m.add_function(wrap_pyfunction!(node::list_children, m)?)?; m.add_function(wrap_pyfunction!(node::read_metadata, m)?)?; Ok(()) } diff --git a/zarrs-bindings/src/node.rs b/zarrs-bindings/src/node.rs index c7833246e6..35a057ab31 100644 --- a/zarrs-bindings/src/node.rs +++ b/zarrs-bindings/src/node.rs @@ -83,3 +83,40 @@ pub(crate) fn read_metadata( serde_json::to_string(node.metadata()).map_err(runtime_err) }) } + +#[pyfunction] +pub(crate) fn delete_node(py: Python<'_>, store: &Bound<'_, PyAny>, path: String) -> PyResult<()> { + let storage = resolve_store(store)?; + py.detach(move || { + let node_path = parse_node_path(&path)?; + if !node_exists(&storage, &node_path).map_err(runtime_err)? { + return Err(NodeNotFoundError::new_err(format!( + "no node found at path {}", + node_path.as_str() + ))); + } + let prefix: StorePrefix = (&node_path).try_into().map_err(value_err)?; + storage.erase_prefix(&prefix).map_err(runtime_err) + }) +} + +#[pyfunction] +pub(crate) fn list_children( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, +) -> PyResult> { + let storage = resolve_store(store)?; + py.detach(move || { + let group = + Group::open(storage, &path).map_err(|e| NodeNotFoundError::new_err(e.to_string()))?; + let children = group.children(false).map_err(runtime_err)?; + children + .into_iter() + .map(|node| { + let metadata = serde_json::to_string(node.metadata()).map_err(runtime_err)?; + Ok((node.path().as_str().to_string(), metadata)) + }) + .collect() + }) +} From dcaf5ef399e3117b6ffeb500b79fa71833d7e91f Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 14:27:48 +0200 Subject: [PATCH 14/75] feat: zarrs-backed whole-chunk decode/encode/raw-read/erase Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/__init__.py | 8 +++ src/zarr/zarrs/_api.py | 118 +++++++++++++++++++++++++++++++- tests/zarrs/test_chunk.py | 130 ++++++++++++++++++++++++++++++++++++ tests/zarrs/test_node.py | 1 + zarrs-bindings/src/chunk.rs | 94 ++++++++++++++++++++++++++ zarrs-bindings/src/lib.rs | 5 ++ 6 files changed, 354 insertions(+), 2 deletions(-) create mode 100644 tests/zarrs/test_chunk.py create mode 100644 zarrs-bindings/src/chunk.rs diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py index 7d4968cd40..d439e9d7c7 100644 --- a/src/zarr/zarrs/__init__.py +++ b/src/zarr/zarrs/__init__.py @@ -27,8 +27,12 @@ create_new_group, create_overwrite_array, create_overwrite_group, + decode_chunk, delete_node, + encode_chunk, + erase_chunk, list_children, + read_encoded_chunk, read_metadata, ) @@ -40,7 +44,11 @@ "create_new_group", "create_overwrite_array", "create_overwrite_group", + "decode_chunk", "delete_node", + "encode_chunk", + "erase_chunk", "list_children", + "read_encoded_chunk", "read_metadata", ] diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index 09c58d6585..8a021faa75 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -4,9 +4,10 @@ import json from contextlib import contextmanager from dataclasses import dataclass -from typing import TYPE_CHECKING, cast +from typing import TYPE_CHECKING, Any, cast import _zarrs_bindings as _zb +import numpy as np from zarr.errors import NodeNotFoundError from zarr.zarrs._bridge import resolve_store @@ -14,6 +15,8 @@ if TYPE_CHECKING: from collections.abc import Iterator, Mapping + import numpy.typing as npt + from zarr.abc.store import Store from zarr.core.common import JSON @@ -157,7 +160,8 @@ async def list_children( `(path, metadata_document)` pairs. Paths are store-relative (no leading `/`). - Raises `zarr.errors.NodeNotFoundError` if no group exists at `path`. + Raises `zarr.errors.NodeNotFoundError` if no *group* exists at `path` -- + including when `path` holds an array. """ with _translate_errors(): raw: list[tuple[str, str]] = await asyncio.to_thread( @@ -167,3 +171,113 @@ async def list_children( (child_path.lstrip("/"), cast("dict[str, JSON]", json.loads(doc))) for child_path, doc in raw ] + + +def _chunk_dtype_and_shape( + metadata: Mapping[str, JSON], +) -> tuple[np.dtype[Any], tuple[int, ...]]: + """Resolve the numpy dtype and chunk shape from a metadata document, using + zarr-python's own metadata parsing.""" + from zarr.core.metadata.v2 import ArrayV2Metadata + from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata + + if metadata.get("zarr_format") == 3: + meta3 = ArrayV3Metadata.from_dict(dict(metadata)) + grid = meta3.chunk_grid + if not isinstance(grid, RegularChunkGridMetadata): + raise NotImplementedError("only regular chunk grids are supported") + return meta3.data_type.to_native_dtype(), grid.chunk_shape + meta2 = ArrayV2Metadata.from_dict(dict(metadata)) + return meta2.dtype.to_native_dtype(), meta2.chunks + + +async def decode_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + selection: tuple[slice | int, ...] | None = None, + options: ZarrsOptions | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode the chunk at `chunk_coords` of the array described by + `metadata`, located at `path` in `store`. + + The metadata document is authoritative: it is not read from the store. + Missing chunks decode to the fill value. `selection` (a chunk-relative + subset) is not implemented yet. + """ + if selection is not None: + raise NotImplementedError("chunk subset selection is not implemented yet") + raw = await asyncio.to_thread( + _zb.retrieve_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + ) + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + return np.frombuffer(raw, dtype=dtype).reshape(chunk_shape) + + +async def read_encoded_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: ZarrsOptions | None = None, +) -> bytes | None: + """Read the raw, still-encoded bytes of the chunk at `chunk_coords`, or + `None` if the chunk does not exist. No codecs are applied.""" + result: bytes | None = await asyncio.to_thread( + _zb.retrieve_encoded_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + ) + return result + + +async def encode_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + value: npt.ArrayLike, + *, + options: ZarrsOptions | None = None, +) -> None: + """Encode `value` with the codecs in `metadata` and store it as the chunk + at `chunk_coords`. `value` must match the chunk shape exactly.""" + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + arr = np.ascontiguousarray(np.asarray(value, dtype=dtype)) + if arr.shape != chunk_shape: + raise ValueError(f"value shape {arr.shape} does not match chunk shape {chunk_shape}") + await asyncio.to_thread( + _zb.store_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + arr.tobytes(), + ) + + +async def erase_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: ZarrsOptions | None = None, +) -> None: + """Delete the chunk at `chunk_coords`. Deleting a missing chunk is a no-op.""" + await asyncio.to_thread( + _zb.erase_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(chunk_coords), + ) diff --git a/tests/zarrs/test_chunk.py b/tests/zarrs/test_chunk.py new file mode 100644 index 0000000000..abcc45a751 --- /dev/null +++ b/tests/zarrs/test_chunk.py @@ -0,0 +1,130 @@ +from __future__ import annotations + +import copy +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +import zarr +from tests.zarrs.conftest import array_metadata +from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec +from zarr.core.buffer.core import default_buffer_prototype +from zarr.zarrs import ( + create_new_array, + decode_chunk, + encode_chunk, + erase_chunk, + read_encoded_chunk, +) + +if TYPE_CHECKING: + from zarr.abc.store import Store + + +def _filled(store: Store, **kwargs: Any) -> tuple[np.ndarray[Any, np.dtype[Any]], dict[str, Any]]: + """Create an 8x8 array named 'a' via zarr-python, fill it with a ramp, and + return (data, metadata_document).""" + params: dict[str, Any] = {"shape": (8, 8), "chunks": (4, 4), "dtype": "uint16"} | kwargs + arr = zarr.create_array(store=store, name="a", **params) + data = np.arange(64, dtype=params["dtype"]).reshape(8, 8) + arr[:, :] = data + doc = dict(arr.metadata.to_dict()) + if params.get("zarr_format") == 2: + # v2 attributes live in .zattrs, not in the .zarray document + doc.pop("attributes", None) + return data, doc + + +@pytest.mark.parametrize("dtype", ["uint8", "int32", "float64"]) +async def test_decode_chunk_differential(store: Store, dtype: str) -> None: + data, meta = _filled(store, dtype=dtype) + observed = await decode_chunk(meta, store, "a", (1, 0)) + np.testing.assert_array_equal(observed, data[4:8, 0:4]) + + +@pytest.mark.parametrize( + "compressors", [None, (GzipCodec(),), (ZstdCodec(),), (BloscCodec(cname="lz4"),)] +) +async def test_decode_chunk_codecs(store: Store, compressors: Any) -> None: + data, meta = _filled(store, compressors=compressors) + observed = await decode_chunk(meta, store, "a", (0, 1)) + np.testing.assert_array_equal(observed, data[0:4, 4:8]) + + +async def test_decode_chunk_v2(store: Store) -> None: + data, meta = _filled(store, zarr_format=2) + observed = await decode_chunk(meta, store, "a", (1, 1)) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_decode_chunk_sharding(store: Store) -> None: + # with sharding, the metadata chunk grid is the shard grid + data, meta = _filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await decode_chunk(meta, store, "a", (1, 1)) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_decode_chunk_missing_returns_fill_value(store: Store) -> None: + arr = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 + ) + meta = dict(arr.metadata.to_dict()) + observed = await decode_chunk(meta, store, "a", (0, 0)) + np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) + + +async def test_decode_chunk_selection_not_implemented(store: Store) -> None: + _, meta = _filled(store) + with pytest.raises(NotImplementedError): + await decode_chunk(meta, store, "a", (0, 0), selection=(slice(0, 2), slice(0, 2))) + + +async def test_decode_chunk_metadata_view(store: Store) -> None: + # the read-only-view case: decode with a metadata document the store never saw + data, meta = _filled(store, dtype="uint16", compressors=None) + view = copy.deepcopy(meta) + view["data_type"] = "uint8" + view["shape"] = [8, 16] + view["chunk_grid"]["configuration"]["chunk_shape"] = [4, 8] + observed = await decode_chunk(view, store, "a", (1, 0)) + np.testing.assert_array_equal(observed, data[4:8, 0:4].view("uint8")) + + +async def test_encode_chunk_differential(store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a") + value = np.arange(16, dtype="uint16").reshape(4, 4) + await encode_chunk(meta, store, "a", (0, 1), value) + arr = zarr.open_array(store=store, path="a", mode="r") + np.testing.assert_array_equal(arr[0:4, 4:8], value) + + +async def test_encode_chunk_shape_mismatch(store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a") + with pytest.raises(ValueError, match="chunk shape"): + await encode_chunk(meta, store, "a", (0, 0), np.zeros((2, 2), dtype="uint16")) + + +async def test_read_encoded_chunk_matches_store(store: Store) -> None: + _, meta = _filled(store) + raw = await read_encoded_chunk(meta, store, "a", (0, 0)) + expected = await store.get("a/c/0/0", prototype=default_buffer_prototype()) + assert expected is not None + assert raw == expected.to_bytes() + + +async def test_read_encoded_chunk_missing_returns_none(store: Store) -> None: + arr = zarr.create_array(store=store, name="empty", shape=(8, 8), chunks=(4, 4), dtype="uint16") + meta = dict(arr.metadata.to_dict()) + assert await read_encoded_chunk(meta, store, "empty", (0, 0)) is None + + +async def test_erase_chunk(store: Store) -> None: + _, meta = _filled(store) + assert await store.exists("a/c/0/0") + await erase_chunk(meta, store, "a", (0, 0)) + assert not await store.exists("a/c/0/0") + arr = zarr.open_array(store=store, path="a", mode="r") + np.testing.assert_array_equal(arr[0:4, 0:4], np.zeros((4, 4), dtype="uint16")) diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py index c5d46f472f..749f71ddf5 100644 --- a/tests/zarrs/test_node.py +++ b/tests/zarrs/test_node.py @@ -129,6 +129,7 @@ async def test_list_children(store: Store) -> None: children = await list_children(store, "") by_path = dict(children) assert set(by_path) == {"sub_group", "sub_array"} + assert not any(p.startswith("/") for p in by_path) assert by_path["sub_group"]["node_type"] == "group" assert by_path["sub_array"]["node_type"] == "array" diff --git a/zarrs-bindings/src/chunk.rs b/zarrs-bindings/src/chunk.rs new file mode 100644 index 0000000000..246a198768 --- /dev/null +++ b/zarrs-bindings/src/chunk.rs @@ -0,0 +1,94 @@ +use pyo3::exceptions::PyNotImplementedError; +use pyo3::prelude::*; +use pyo3::types::PyBytes; +use zarrs::array::{Array, ArrayBytes}; +use zarrs::metadata::ArrayMetadata; +use zarrs::storage::ReadableWritableListableStorage; + +use crate::store::resolve_store; +use crate::{runtime_err, value_err}; + +type DynArray = Array; + +/// Construct an Array view from an explicit metadata document, without +/// consulting the store for metadata. +fn array_view( + storage: ReadableWritableListableStorage, + path: &str, + metadata_json: &str, +) -> PyResult { + let metadata = ArrayMetadata::try_from(metadata_json).map_err(value_err)?; + Array::new_with_metadata(storage, path, metadata).map_err(value_err) +} + +#[pyfunction] +pub(crate) fn retrieve_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, +) -> PyResult> { + let storage = resolve_store(store)?; + let data = py.detach(move || -> PyResult> { + let array = array_view(storage, &path, &metadata_json)?; + let bytes: ArrayBytes<'static> = + array.retrieve_chunk(&chunk_coords).map_err(runtime_err)?; + let fixed = bytes.into_fixed().map_err(|_| { + PyNotImplementedError::new_err("variable-length data types are not supported") + })?; + Ok(fixed.into_owned()) + })?; + Ok(PyBytes::new(py, &data).unbind()) +} + +#[pyfunction] +pub(crate) fn retrieve_encoded_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, +) -> PyResult>> { + let storage = resolve_store(store)?; + let data = py.detach(move || -> PyResult>> { + let array = array_view(storage, &path, &metadata_json)?; + array + .retrieve_encoded_chunk(&chunk_coords) + .map_err(runtime_err) + })?; + Ok(data.map(|d| PyBytes::new(py, &d).unbind())) +} + +#[pyfunction] +pub(crate) fn store_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, + data: Vec, +) -> PyResult<()> { + let storage = resolve_store(store)?; + py.detach(move || { + let array = array_view(storage, &path, &metadata_json)?; + array + .store_chunk(&chunk_coords, ArrayBytes::new_flen(data)) + .map_err(runtime_err) + }) +} + +#[pyfunction] +pub(crate) fn erase_chunk( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + chunk_coords: Vec, +) -> PyResult<()> { + let storage = resolve_store(store)?; + py.detach(move || { + let array = array_view(storage, &path, &metadata_json)?; + array.erase_chunk(&chunk_coords).map_err(runtime_err) + }) +} diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs index 6a98d4ec53..2e62b6538f 100644 --- a/zarrs-bindings/src/lib.rs +++ b/zarrs-bindings/src/lib.rs @@ -1,6 +1,7 @@ use pyo3::exceptions::{PyRuntimeError, PyValueError}; use pyo3::prelude::*; +mod chunk; mod node; mod store; @@ -40,5 +41,9 @@ fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(node::delete_node, m)?)?; m.add_function(wrap_pyfunction!(node::list_children, m)?)?; m.add_function(wrap_pyfunction!(node::read_metadata, m)?)?; + m.add_function(wrap_pyfunction!(chunk::retrieve_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::retrieve_encoded_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::store_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::erase_chunk, m)?)?; Ok(()) } From 63429b5be111879c4004d22d8355e04d8add6ab3 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 18:38:23 +0200 Subject: [PATCH 15/75] fix: coerce chunk dtype to native byte order for zarrs I/O Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_api.py | 14 +++++++++++--- tests/zarrs/test_chunk.py | 21 +++++++++++++++++++++ 2 files changed, 32 insertions(+), 3 deletions(-) diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index 8a021faa75..318c39348c 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -177,7 +177,12 @@ def _chunk_dtype_and_shape( metadata: Mapping[str, JSON], ) -> tuple[np.dtype[Any], tuple[int, ...]]: """Resolve the numpy dtype and chunk shape from a metadata document, using - zarr-python's own metadata parsing.""" + zarr-python's own metadata parsing. + + The dtype is coerced to native byte order: zarrs always decodes to (and + encodes from) the native in-memory representation, applying any byte-order + codec itself. + """ from zarr.core.metadata.v2 import ArrayV2Metadata from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata @@ -186,9 +191,9 @@ def _chunk_dtype_and_shape( grid = meta3.chunk_grid if not isinstance(grid, RegularChunkGridMetadata): raise NotImplementedError("only regular chunk grids are supported") - return meta3.data_type.to_native_dtype(), grid.chunk_shape + return meta3.data_type.to_native_dtype().newbyteorder("="), grid.chunk_shape meta2 = ArrayV2Metadata.from_dict(dict(metadata)) - return meta2.dtype.to_native_dtype(), meta2.chunks + return meta2.dtype.to_native_dtype().newbyteorder("="), meta2.chunks async def decode_chunk( @@ -206,6 +211,9 @@ async def decode_chunk( The metadata document is authoritative: it is not read from the store. Missing chunks decode to the fill value. `selection` (a chunk-relative subset) is not implemented yet. + + The returned array is a read-only, zero-copy view over the decoded bytes; + call `.copy()` if you need a writable array. """ if selection is not None: raise NotImplementedError("chunk subset selection is not implemented yet") diff --git a/tests/zarrs/test_chunk.py b/tests/zarrs/test_chunk.py index abcc45a751..64456fe954 100644 --- a/tests/zarrs/test_chunk.py +++ b/tests/zarrs/test_chunk.py @@ -58,6 +58,27 @@ async def test_decode_chunk_v2(store: Store) -> None: np.testing.assert_array_equal(observed, data[4:8, 4:8]) +async def test_decode_chunk_v2_big_endian(store: Store) -> None: + data, meta = _filled(store, dtype=">u2", zarr_format=2) + observed = await decode_chunk(meta, store, "a", (1, 1)) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_encode_chunk_v2_big_endian(store: Store) -> None: + meta = array_metadata(dtype=">u2", zarr_format=2) + await create_new_array(meta, store, "a") + value = np.arange(16, dtype="uint16").reshape(4, 4) + await encode_chunk(meta, store, "a", (0, 1), value) + arr = zarr.open_array(store=store, path="a", mode="r") + np.testing.assert_array_equal(arr[0:4, 4:8], value) + + +async def test_decode_chunk_readonly(store: Store) -> None: + _, meta = _filled(store) + observed = await decode_chunk(meta, store, "a", (0, 0)) + assert not observed.flags.writeable + + async def test_decode_chunk_sharding(store: Store) -> None: # with sharding, the metadata chunk grid is the shard grid data, meta = _filled(store, chunks=(2, 2), shards=(4, 4)) From e5dc4825852cc22578589de359012219b9d8186d Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 18:41:17 +0200 Subject: [PATCH 16/75] chore: lint fixes and changelog for zarr.zarrs Co-Authored-By: Claude Fable 5 --- changes/+zarrs-bindings.feature.md | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 changes/+zarrs-bindings.feature.md diff --git a/changes/+zarrs-bindings.feature.md b/changes/+zarrs-bindings.feature.md new file mode 100644 index 0000000000..0b0e7ee384 --- /dev/null +++ b/changes/+zarrs-bindings.feature.md @@ -0,0 +1,6 @@ +Added `zarr.zarrs`, an experimental low-level functional API for zarr hierarchy +CRUD backed by the Rust [zarrs](https://zarrs.dev) crate via the new in-repo +`zarrs-bindings` PyO3 crate. Array routines take an explicit metadata document, +enabling read-only views such as decoding chunks with externally supplied +metadata or reading raw encoded chunk bytes. Build for development with +`uv sync --group zarrs`. From 9dd5abdbad8a4e4690deabd055452c2f7704c36b Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 18:50:51 +0200 Subject: [PATCH 17/75] ci: test job for zarrs bindings Co-Authored-By: Claude Fable 5 --- .github/workflows/zarrs.yml | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 .github/workflows/zarrs.yml diff --git a/.github/workflows/zarrs.yml b/.github/workflows/zarrs.yml new file mode 100644 index 0000000000..2df419772d --- /dev/null +++ b/.github/workflows/zarrs.yml @@ -0,0 +1,32 @@ +name: Zarrs bindings + +on: + push: + branches: [ main ] + pull_request: + branches: [ main ] + workflow_dispatch: + +permissions: + contents: read + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + fetch-depth: 0 # hatch-vcs needs tags to compute zarr's version + persist-credentials: false + - name: Install uv + uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + with: + python-version: '3.12' + - name: Run zarrs bindings tests + # the ubuntu runner image ships a Rust toolchain; the maturin build + # backend is fetched by uv on demand + run: uv run --group zarrs pytest tests/zarrs -v From 7226b8d4914848334ff2068becb9bcefdbcdc94c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 20:11:14 +0200 Subject: [PATCH 18/75] fix: keep zarr.zarrs out of doctest collection; pin Rust toolchain in CI - Add --ignore=src/zarr/zarrs to pytest addopts so --doctest-modules doesn't attempt to import the bindings module when zarrs-bindings is not installed, preventing a collection ERROR in jobs that don't use the zarrs dependency group. - Add dtolnay/rust-toolchain step (SHA-pinned, stable) to zarrs.yml CI so the build is not reliant on whatever Rust version the runner image ships; ensures rust-version = "1.91" in the crate is satisfied. - Fix spec: abi3-py311 -> abi3-py312 to match zarr's requires-python >=3.12. Co-Authored-By: Claude Fable 5 --- .github/workflows/zarrs.yml | 2 ++ .../superpowers/specs/2026-06-11-zarrs-functional-api-design.md | 2 +- pyproject.toml | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/zarrs.yml b/.github/workflows/zarrs.yml index 2df419772d..954dcda79b 100644 --- a/.github/workflows/zarrs.yml +++ b/.github/workflows/zarrs.yml @@ -26,6 +26,8 @@ jobs: uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 with: python-version: '3.12' + - name: Install Rust + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable - name: Run zarrs bindings tests # the ubuntu runner image ships a Rust toolchain; the maturin build # backend is fetched by uv on demand diff --git a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md index ebf2854611..091c8cb211 100644 --- a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md +++ b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md @@ -40,7 +40,7 @@ codec-pipeline registry through this API (possible later), fancy Two distributions in this repo, hard boundary between them: 1. **Rust crate `zarrs-bindings`** at the repo root (`zarrs-bindings/`), - built with maturin (PyO3, `abi3-py311`), publishing wheel `zarrs-bindings` + built with maturin (PyO3, `abi3-py312`), publishing wheel `zarrs-bindings` with native module `_zarrs_bindings`. It is a thin, mechanical binding over `zarrs`: functions/pyclasses take metadata as a **JSON string**, a store-config object, a node path, and return bytes / numpy arrays. It knows diff --git a/pyproject.toml b/pyproject.toml index 90368ed37a..d24917bc5f 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -441,6 +441,7 @@ addopts = [ "--doctest-modules", "--ignore=tests/test_regression/scripts", "--ignore=src/zarr/_cli", + "--ignore=src/zarr/zarrs", ] filterwarnings = [ "error", From 2b2a6b5692342f7535aed9d331f117a2489a1aeb Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 21:35:48 +0200 Subject: [PATCH 19/75] fix: make zarrs test skip xdist-compatible Move the pytest.importorskip("_zarrs_bindings") guard from conftest.py (module-level) to the top of each test module. When the bindings are absent, xdist workers would previously raise Skipped while importing the conftest, causing "Different tests were collected between gw0 and gwX" failures. Per-module guards are the standard xdist-safe pattern. Co-Authored-By: Claude Fable 5 --- tests/zarrs/conftest.py | 4 ---- tests/zarrs/test_api.py | 6 ++++++ tests/zarrs/test_bridge.py | 6 ++++++ tests/zarrs/test_chunk.py | 4 ++++ tests/zarrs/test_node.py | 4 ++++ 5 files changed, 20 insertions(+), 4 deletions(-) diff --git a/tests/zarrs/conftest.py b/tests/zarrs/conftest.py index 678065e5e1..092bce5473 100644 --- a/tests/zarrs/conftest.py +++ b/tests/zarrs/conftest.py @@ -4,10 +4,6 @@ import pytest -pytest.importorskip( - "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError -) - import zarr from zarr.storage import LocalStore, MemoryStore diff --git a/tests/zarrs/test_api.py b/tests/zarrs/test_api.py index da1a9ecda8..1a3e9005e2 100644 --- a/tests/zarrs/test_api.py +++ b/tests/zarrs/test_api.py @@ -1,5 +1,11 @@ from __future__ import annotations +import pytest + +pytest.importorskip( + "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError +) + def test_import() -> None: import zarr.zarrs diff --git a/tests/zarrs/test_bridge.py b/tests/zarrs/test_bridge.py index 2b88c047dd..f997b052f2 100644 --- a/tests/zarrs/test_bridge.py +++ b/tests/zarrs/test_bridge.py @@ -2,6 +2,12 @@ from typing import TYPE_CHECKING +import pytest + +pytest.importorskip( + "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError +) + from zarr.storage import LocalStore, MemoryStore from zarr.zarrs._bridge import StoreShim, resolve_store diff --git a/tests/zarrs/test_chunk.py b/tests/zarrs/test_chunk.py index 64456fe954..74a494770f 100644 --- a/tests/zarrs/test_chunk.py +++ b/tests/zarrs/test_chunk.py @@ -6,6 +6,10 @@ import numpy as np import pytest +pytest.importorskip( + "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError +) + import zarr from tests.zarrs.conftest import array_metadata from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py index 749f71ddf5..eefb4cf137 100644 --- a/tests/zarrs/test_node.py +++ b/tests/zarrs/test_node.py @@ -6,6 +6,10 @@ import numpy as np import pytest +pytest.importorskip( + "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError +) + import zarr from tests.zarrs.conftest import array_metadata from zarr.core.buffer.core import default_buffer_prototype From 100a7cbc33320dd70533befceb1f8b90b931a250 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Fri, 12 Jun 2026 22:19:14 +0200 Subject: [PATCH 20/75] feat: decode_region with numpy-style basic indexing Add `retrieve_array_subset` Rust binding and `decode_region` Python API function. Selection normaliser maps integers/slices/Ellipsis to a step-1 bounding box fetched in one zarrs call; strides, reversals, and integer-axis removal are applied as numpy views on the result. Co-Authored-By: Claude Fable 5 --- .../2026-06-11-zarrs-functional-api-design.md | 7 +- src/zarr/zarrs/__init__.py | 2 + src/zarr/zarrs/_api.py | 133 ++++++++++++++++++ tests/zarrs/test_chunk.py | 69 +++++++++ zarrs-bindings/src/chunk.rs | 25 +++- zarrs-bindings/src/lib.rs | 1 + 6 files changed, 233 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md index 091c8cb211..bfafd59aae 100644 --- a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md +++ b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md @@ -65,7 +65,7 @@ All functions are `async def`. Parameters: - `store`: `zarr.abc.store.Store`. - `path`: node path within the store (str, `""` = root). - `chunk_coords`: `tuple[int, ...]` grid coordinates. -- `selection`: tuple of `slice`/`int` only (v1 restriction). +- `selection`: numpy-style basic indexing — integers, slices (including steps; strided/reversed selections fetch the step-1 bounding box in one call and apply numpy views), and `Ellipsis`. Fancy indexing (integer/boolean arrays) and `np.newaxis` are not supported. - `options`: every function also accepts keyword-only `options: ZarrsOptions | None = None` (omitted from the signatures below for brevity) — a dataclass holding concurrency limits and checksum validation @@ -170,8 +170,9 @@ propagation of store-callback exceptions (and richer mapping onto 1. **Phase 1**: crate scaffolding (maturin, CI build), store bridge (native LocalStore + generic PyStore), node lifecycle functions, whole-chunk `decode_chunk` / `read_encoded_chunk` / `encode_chunk` / `erase_chunk`. -2. **Phase 2**: `decode_region` / `encode_region`, chunk-subset `selection` - via partial decoders. +2. **Phase 2**: `decode_region` (read side of region I/O) is implemented on + this branch. `encode_region` and chunk-subset `selection` for `decode_chunk` + via partial decoders remain Phase 2. 3. **Phase 3**: `ZarrsOptions` surface (concurrency, checksum validation, direct IO), obstore native path, benchmarks vs. the pure-Python pipeline. diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py index d439e9d7c7..1e287b9cfa 100644 --- a/src/zarr/zarrs/__init__.py +++ b/src/zarr/zarrs/__init__.py @@ -28,6 +28,7 @@ create_overwrite_array, create_overwrite_group, decode_chunk, + decode_region, delete_node, encode_chunk, erase_chunk, @@ -45,6 +46,7 @@ "create_overwrite_array", "create_overwrite_group", "decode_chunk", + "decode_region", "delete_node", "encode_chunk", "erase_chunk", diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index 318c39348c..d97acd0984 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -2,6 +2,9 @@ import asyncio import json +import operator +import types +from collections.abc import Sequence from contextlib import contextmanager from dataclasses import dataclass from typing import TYPE_CHECKING, Any, cast @@ -33,6 +36,10 @@ class ZarrsOptions: """ +BasicIndex = int | slice | types.EllipsisType +BasicSelection = BasicIndex | tuple[BasicIndex, ...] + + def _node_path(path: str) -> str: """Convert a zarr-python node path (`""`, `"foo/bar"`) to a zarrs node path (`"/"`, `"/foo/bar"`).""" @@ -173,6 +180,93 @@ async def list_children( ] +def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: + """Resolve the array shape from a metadata document.""" + shape = metadata.get("shape") + if not isinstance(shape, Sequence) or isinstance(shape, str): + raise TypeError("metadata document has no valid 'shape'") + result: list[int] = [] + for s in shape: + if not isinstance(s, (int, float)): + raise TypeError(f"shape element {s!r} is not a number") + result.append(int(s)) + return tuple(result) + + +def _normalize_selection( + selection: BasicSelection, shape: tuple[int, ...] +) -> tuple[list[int], list[int], tuple[slice | int, ...]]: + """Normalize a numpy-style basic-indexing selection against `shape`. + + Returns `(start, bounding_shape, post_index)`: the step-1 bounding box to + fetch (per-dimension start and length), and the numpy index to apply to + the fetched block to produce the final result (strides, reversals, and + integer-axis removal). Only integers, slices, and `Ellipsis` are + supported; fancy indexing raises `TypeError`. + """ + sel_tuple = selection if isinstance(selection, tuple) else (selection,) + + n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) + if n_ellipsis > 1: + raise IndexError("an index can only have a single ellipsis ('...')") + if n_ellipsis == 1: + i = sel_tuple.index(Ellipsis) + n_fill = len(shape) - (len(sel_tuple) - 1) + if n_fill < 0: + raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") + sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] + if len(sel_tuple) > len(shape): + raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") + sel_tuple = sel_tuple + (slice(None),) * (len(shape) - len(sel_tuple)) + + starts: list[int] = [] + lengths: list[int] = [] + post: list[slice | int] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + start, stop, step = sel.indices(size) + n = len(range(start, stop, step)) + if n == 0: + starts.append(0) + lengths.append(0) + post.append(slice(None)) + elif step > 0: + last = start + (n - 1) * step + starts.append(start) + lengths.append(last - start + 1) + post.append(slice(None, None, step)) + else: + # descending: bounding box is [last, start], ascending in store + # order; slice(None, None, step) over the block starts at its + # final element (global `start`) and lands exactly on index 0 + # (global `last`) because the block length is (n-1)*|step| + 1. + last = start + (n - 1) * step + starts.append(last) + lengths.append(start - last + 1) + post.append(slice(None, None, step)) + else: + if isinstance(sel, types.EllipsisType): + raise TypeError( + "unsupported selection element " + f"{sel!r}: only integers, slices, and Ellipsis are supported" + ) + try: + idx = operator.index(sel) + except TypeError: + raise TypeError( + "unsupported selection element " + f"{sel!r}: only integers, slices, and Ellipsis are supported" + ) from None + if idx < 0: + idx += size + if not 0 <= idx < size: + raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") + starts.append(idx) + lengths.append(1) + post.append(0) + return starts, lengths, tuple(post) + + def _chunk_dtype_and_shape( metadata: Mapping[str, JSON], ) -> tuple[np.dtype[Any], tuple[int, ...]]: @@ -289,3 +383,42 @@ async def erase_chunk( json.dumps(metadata), list(chunk_coords), ) + + +async def decode_region( + metadata: Mapping[str, JSON], + store: Store, + path: str, + selection: BasicSelection, + *, + options: ZarrsOptions | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode the region of the array described by `metadata` given + by a numpy-style basic-indexing `selection` (integers, slices including + steps, `Ellipsis`). + + The metadata document is authoritative: it is not read from the store. + One zarrs call fetches the step-1 bounding box of the selection (decoding + all overlapping chunks, in parallel for multi-chunk regions); strides, + reversals, and integer-axis removal are applied as numpy views on the + result. Missing chunks decode to the fill value. Fancy indexing (integer + or boolean arrays) is not supported and raises `TypeError`. The returned + array is a read-only view; call `.copy()` if you need a writable array. + """ + dtype, _ = _chunk_dtype_and_shape(metadata) + shape = _array_shape(metadata) + starts, lengths, post_index = _normalize_selection(selection, shape) + if 0 in lengths: + block = np.empty(lengths, dtype=dtype) + else: + raw = await asyncio.to_thread( + _zb.retrieve_array_subset, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + starts, + lengths, + ) + block = np.frombuffer(raw, dtype=dtype).reshape(lengths) + result: np.ndarray[Any, np.dtype[Any]] = block[post_index] + return result diff --git a/tests/zarrs/test_chunk.py b/tests/zarrs/test_chunk.py index 74a494770f..89cf10304d 100644 --- a/tests/zarrs/test_chunk.py +++ b/tests/zarrs/test_chunk.py @@ -17,6 +17,7 @@ from zarr.zarrs import ( create_new_array, decode_chunk, + decode_region, encode_chunk, erase_chunk, read_encoded_chunk, @@ -153,3 +154,71 @@ async def test_erase_chunk(store: Store) -> None: assert not await store.exists("a/c/0/0") arr = zarr.open_array(store=store, path="a", mode="r") np.testing.assert_array_equal(arr[0:4, 0:4], np.zeros((4, 4), dtype="uint16")) + + +SELECTIONS: list[Any] = [ + (slice(None), slice(None)), + (slice(2, 7), slice(1, 5)), # crosses chunk boundaries + (slice(None), 3), + (5, slice(None)), + (3, 4), # fully scalar -> 0-d result + (slice(1, 8, 2), slice(None)), + (slice(None), slice(6, 1, -2)), # negative step + (slice(-3, None), slice(None, -1)), # negative bounds + ..., # Ellipsis alone + (..., slice(2, 4)), + (slice(0, 0), slice(None)), # empty + (slice(2, 6),), # partial selection, missing trailing dims +] + + +@pytest.mark.parametrize("sel", SELECTIONS) +async def test_decode_region_differential(store: Store, sel: Any) -> None: + data, meta = _filled(store) + observed = await decode_region(meta, store, "a", sel) + np.testing.assert_array_equal(observed, data[sel]) + + +async def test_decode_region_sharding(store: Store) -> None: + data, meta = _filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await decode_region(meta, store, "a", (slice(1, 7), slice(3, 8))) + np.testing.assert_array_equal(observed, data[1:7, 3:8]) + + +async def test_decode_region_v2(store: Store) -> None: + data, meta = _filled(store, zarr_format=2) + observed = await decode_region(meta, store, "a", (slice(2, 7), slice(None, None, 3))) + np.testing.assert_array_equal(observed, data[2:7, ::3]) + + +async def test_decode_region_missing_chunks_fill_value(store: Store) -> None: + arr = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 + ) + meta = dict(arr.metadata.to_dict()) + observed = await decode_region(meta, store, "a", (slice(2, 6), slice(2, 6))) + np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) + + +async def test_decode_region_out_of_bounds(store: Store) -> None: + _, meta = _filled(store) + with pytest.raises(IndexError, match="out of bounds"): + await decode_region(meta, store, "a", (8, slice(None))) + + +async def test_decode_region_too_many_indices(store: Store) -> None: + _, meta = _filled(store) + with pytest.raises(IndexError, match="too many indices"): + await decode_region(meta, store, "a", (0, 0, 0)) + + +async def test_decode_region_fancy_indexing_rejected(store: Store) -> None: + _, meta = _filled(store) + with pytest.raises(TypeError, match="only integers, slices"): + await decode_region(meta, store, "a", ([0, 1], slice(None))) # type: ignore[arg-type] + + +async def test_decode_region_readonly(store: Store) -> None: + _, meta = _filled(store) + observed = await decode_region(meta, store, "a", (slice(0, 4), slice(0, 4))) + assert not observed.flags.writeable diff --git a/zarrs-bindings/src/chunk.rs b/zarrs-bindings/src/chunk.rs index 246a198768..30c0b04554 100644 --- a/zarrs-bindings/src/chunk.rs +++ b/zarrs-bindings/src/chunk.rs @@ -1,7 +1,7 @@ use pyo3::exceptions::PyNotImplementedError; use pyo3::prelude::*; use pyo3::types::PyBytes; -use zarrs::array::{Array, ArrayBytes}; +use zarrs::array::{Array, ArrayBytes, ArraySubset}; use zarrs::metadata::ArrayMetadata; use zarrs::storage::ReadableWritableListableStorage; @@ -92,3 +92,26 @@ pub(crate) fn erase_chunk( array.erase_chunk(&chunk_coords).map_err(runtime_err) }) } + +#[pyfunction] +pub(crate) fn retrieve_array_subset( + py: Python<'_>, + store: &Bound<'_, PyAny>, + path: String, + metadata_json: String, + start: Vec, + shape: Vec, +) -> PyResult> { + let storage = resolve_store(store)?; + let data = py.detach(move || -> PyResult> { + let array = array_view(storage, &path, &metadata_json)?; + let subset = ArraySubset::new_with_start_shape(start, shape).map_err(value_err)?; + let bytes: ArrayBytes<'static> = + array.retrieve_array_subset(&subset).map_err(runtime_err)?; + let fixed = bytes.into_fixed().map_err(|_| { + PyNotImplementedError::new_err("variable-length data types are not supported") + })?; + Ok(fixed.into_owned()) + })?; + Ok(PyBytes::new(py, &data).unbind()) +} diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs index 2e62b6538f..aa4552c897 100644 --- a/zarrs-bindings/src/lib.rs +++ b/zarrs-bindings/src/lib.rs @@ -45,5 +45,6 @@ fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(chunk::retrieve_encoded_chunk, m)?)?; m.add_function(wrap_pyfunction!(chunk::store_chunk, m)?)?; m.add_function(wrap_pyfunction!(chunk::erase_chunk, m)?)?; + m.add_function(wrap_pyfunction!(chunk::retrieve_array_subset, m)?)?; Ok(()) } From ccbf9624169b6f20b5c298de99bdcd82fdc832c0 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Sat, 13 Jun 2026 23:57:36 +0200 Subject: [PATCH 21/75] fix: keep empty-selection decode_region result read-only Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_api.py | 1 + tests/zarrs/test_chunk.py | 2 ++ 2 files changed, 3 insertions(+) diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index d97acd0984..40efe13af8 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -410,6 +410,7 @@ async def decode_region( starts, lengths, post_index = _normalize_selection(selection, shape) if 0 in lengths: block = np.empty(lengths, dtype=dtype) + block.flags.writeable = False else: raw = await asyncio.to_thread( _zb.retrieve_array_subset, diff --git a/tests/zarrs/test_chunk.py b/tests/zarrs/test_chunk.py index 89cf10304d..6b4964413c 100644 --- a/tests/zarrs/test_chunk.py +++ b/tests/zarrs/test_chunk.py @@ -222,3 +222,5 @@ async def test_decode_region_readonly(store: Store) -> None: _, meta = _filled(store) observed = await decode_region(meta, store, "a", (slice(0, 4), slice(0, 4))) assert not observed.flags.writeable + empty = await decode_region(meta, store, "a", (slice(0, 0), slice(None))) + assert not empty.flags.writeable From 8895224aac6a3087d5c8a624f3ba532fffbdaa5b Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 09:55:15 +0200 Subject: [PATCH 22/75] docs: note decode_region overread; tidy selection normalizer - Add docstring note to decode_region warning that zarrs fetches the step-1 bounding box, so strided selections read O(span) bytes. - Remove dead isinstance(sel, EllipsisType) raise in _normalize_selection (Ellipsis is expanded to slice(None) before the per-dimension loop); replace with an assert to preserve mypy type narrowing. - Guard non-integral float shape elements in _array_shape so shape=[1.5] raises TypeError instead of silently truncating to 1. Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_api.py | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py index 40efe13af8..df0804f719 100644 --- a/src/zarr/zarrs/_api.py +++ b/src/zarr/zarrs/_api.py @@ -189,6 +189,8 @@ def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: for s in shape: if not isinstance(s, (int, float)): raise TypeError(f"shape element {s!r} is not a number") + if isinstance(s, float) and not s.is_integer(): + raise TypeError(f"shape element {s!r} is not an integer") result.append(int(s)) return tuple(result) @@ -245,11 +247,7 @@ def _normalize_selection( lengths.append(start - last + 1) post.append(slice(None, None, step)) else: - if isinstance(sel, types.EllipsisType): - raise TypeError( - "unsupported selection element " - f"{sel!r}: only integers, slices, and Ellipsis are supported" - ) + assert not isinstance(sel, types.EllipsisType), "Ellipsis already expanded above" try: idx = operator.index(sel) except TypeError: @@ -404,6 +402,11 @@ async def decode_region( result. Missing chunks decode to the fill value. Fancy indexing (integer or boolean arrays) is not supported and raises `TypeError`. The returned array is a read-only view; call `.copy()` if you need a writable array. + + Note: zarrs fetches the step-1 bounding box of the selection. A selection + like `slice(0, N, step)` reads `O(N)` bytes from the store even though only + `O(N / step)` are returned; for sparse selections over large arrays, prefer + reading per-chunk with `decode_chunk`. """ dtype, _ = _chunk_dtype_and_shape(metadata) shape = _array_shape(metadata) From 5faf19846f26e5bd5506dd9950a0c959e57892fa Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 10:00:23 +0200 Subject: [PATCH 23/75] perf: cache constructed Arrays for native filesystem stores Add a process-wide LRU cache (capacity 128) in chunk.rs keyed on (filesystem root, node path, metadata JSON) that memoises the result of Array::new_with_metadata for the native FilesystemStore path. Generic Python-backed stores (MemoryStore, ZipStore, custom) are not cached. The cache key encodes root + path + metadata so an Array is never reused for a different store or different codec chain; chunk data continues to flow through the store on every call, so the cache cannot return stale data. Two test-hook pyfunctions (array_cache_len / clear_array_cache) are exposed on _zarrs_bindings; five correctness tests in tests/zarrs/test_cache.py cover population, non-caching of MemoryStore, distinct-metadata entries, root-keying, and write visibility. All 117 tests pass; cargo clippy -D warnings clean. Co-Authored-By: Claude Fable 5 --- tests/zarrs/test_cache.py | 86 +++++++++++++++++++++++++++++++++++++ zarrs-bindings/Cargo.lock | 14 +++++- zarrs-bindings/Cargo.toml | 1 + zarrs-bindings/src/chunk.rs | 72 +++++++++++++++++++++++++------ zarrs-bindings/src/lib.rs | 2 + zarrs-bindings/src/store.rs | 21 ++++++--- 6 files changed, 175 insertions(+), 21 deletions(-) create mode 100644 tests/zarrs/test_cache.py diff --git a/tests/zarrs/test_cache.py b/tests/zarrs/test_cache.py new file mode 100644 index 0000000000..41e3e24f29 --- /dev/null +++ b/tests/zarrs/test_cache.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +pytest.importorskip( + "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError +) + +import _zarrs_bindings as zb + +import zarr +from zarr.storage import LocalStore, MemoryStore +from zarr.zarrs import decode_chunk, encode_chunk + +if TYPE_CHECKING: + from pathlib import Path + + +def _meta(store: Any, name: str = "a") -> dict[str, Any]: + arr = zarr.create_array(store=store, name=name, shape=(8, 8), chunks=(4, 4), dtype="uint16") + arr[:, :] = np.arange(64, dtype="uint16").reshape(8, 8) + return dict(arr.metadata.to_dict()) + + +@pytest.fixture(autouse=True) +def _clear_cache() -> None: + zb.clear_array_cache() + + +async def test_localstore_populates_cache(tmp_path: Path) -> None: + store = await LocalStore.open(root=tmp_path / "s") + meta = _meta(store) + assert zb.array_cache_len() == 0 + await decode_chunk(meta, store, "a", (0, 0)) + assert zb.array_cache_len() == 1 + # second op on the SAME array reuses the entry, does not grow the cache + await decode_chunk(meta, store, "a", (1, 1)) + assert zb.array_cache_len() == 1 + + +async def test_memorystore_is_not_cached() -> None: + store = MemoryStore() + meta = _meta(store) + await decode_chunk(meta, store, "a", (0, 0)) + assert zb.array_cache_len() == 0 + + +async def test_distinct_metadata_distinct_entries(tmp_path: Path) -> None: + store = await LocalStore.open(root=tmp_path / "s") + meta_a = _meta(store, "a") + meta_b = _meta(store, "b") + await decode_chunk(meta_a, store, "a", (0, 0)) + await decode_chunk(meta_b, store, "b", (0, 0)) + assert zb.array_cache_len() == 2 + + +async def test_cache_keyed_on_root_not_just_metadata(tmp_path: Path) -> None: + # two stores at different roots, identical metadata + path, different data. + # A correct cache (keyed on root) must return each store's own data. + s1 = await LocalStore.open(root=tmp_path / "s1") + s2 = await LocalStore.open(root=tmp_path / "s2") + a1 = zarr.create_array(store=s1, name="a", shape=(4, 4), chunks=(4, 4), dtype="uint16") + a1[:, :] = 1 + a2 = zarr.create_array(store=s2, name="a", shape=(4, 4), chunks=(4, 4), dtype="uint16") + a2[:, :] = 2 + meta = dict(a1.metadata.to_dict()) # identical metadata document + out1 = await decode_chunk(meta, s1, "a", (0, 0)) + out2 = await decode_chunk(meta, s2, "a", (0, 0)) + np.testing.assert_array_equal(out1, np.full((4, 4), 1, dtype="uint16")) + np.testing.assert_array_equal(out2, np.full((4, 4), 2, dtype="uint16")) + assert zb.array_cache_len() == 2 + + +async def test_cache_reflects_writes_through_store(tmp_path: Path) -> None: + # after the Array is cached, a write via the cached Array must be visible to + # a subsequent read (proves the cache does not stale-cache chunk data) + store = await LocalStore.open(root=tmp_path / "s") + meta = _meta(store) + await decode_chunk(meta, store, "a", (0, 0)) # caches the Array + new = np.full((4, 4), 99, dtype="uint16") + await encode_chunk(meta, store, "a", (0, 0), new) # write via (cached) Array + out = await decode_chunk(meta, store, "a", (0, 0)) + np.testing.assert_array_equal(out, new) diff --git a/zarrs-bindings/Cargo.lock b/zarrs-bindings/Cargo.lock index a86d4e26e2..9c9a91f203 100644 --- a/zarrs-bindings/Cargo.lock +++ b/zarrs-bindings/Cargo.lock @@ -468,6 +468,8 @@ version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ + "allocator-api2", + "equivalent", "foldhash 0.1.5", ] @@ -605,6 +607,15 @@ version = "0.4.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" +[[package]] +name = "lru" +version = "0.12.5" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38" +dependencies = [ + "hashbrown 0.15.5", +] + [[package]] name = "lru" version = "0.16.4" @@ -1585,7 +1596,7 @@ dependencies = [ "itoa", "libz-sys", "log", - "lru", + "lru 0.16.4", "moka", "ndarray", "num", @@ -1616,6 +1627,7 @@ dependencies = [ name = "zarrs-bindings" version = "0.1.0" dependencies = [ + "lru 0.12.5", "pyo3", "serde_json", "zarrs", diff --git a/zarrs-bindings/Cargo.toml b/zarrs-bindings/Cargo.toml index e0f381f416..ab06ef3517 100644 --- a/zarrs-bindings/Cargo.toml +++ b/zarrs-bindings/Cargo.toml @@ -12,6 +12,7 @@ name = "_zarrs_bindings" crate-type = ["cdylib"] [dependencies] +lru = "0.12" pyo3 = { version = "0.28", features = ["abi3-py312"] } serde_json = "1" zarrs = "0.23" diff --git a/zarrs-bindings/src/chunk.rs b/zarrs-bindings/src/chunk.rs index 30c0b04554..7d77f10512 100644 --- a/zarrs-bindings/src/chunk.rs +++ b/zarrs-bindings/src/chunk.rs @@ -1,3 +1,7 @@ +use std::num::NonZeroUsize; +use std::sync::{Arc, Mutex, OnceLock}; + +use lru::LruCache; use pyo3::exceptions::PyNotImplementedError; use pyo3::prelude::*; use pyo3::types::PyBytes; @@ -5,14 +9,22 @@ use zarrs::array::{Array, ArrayBytes, ArraySubset}; use zarrs::metadata::ArrayMetadata; use zarrs::storage::ReadableWritableListableStorage; -use crate::store::resolve_store; +use crate::store::resolve_store_with_key; use crate::{runtime_err, value_err}; type DynArray = Array; -/// Construct an Array view from an explicit metadata document, without -/// consulting the store for metadata. -fn array_view( +/// Cache of constructed Arrays keyed by (filesystem root, node path, metadata +/// JSON). Only native filesystem stores are cached (see `resolve_store_with_key`). +/// Bounded by an LRU; entries hold only a filesystem path + codec chain, no data. +type CacheKey = (String, String, String); +static ARRAY_CACHE: OnceLock>>> = OnceLock::new(); + +fn array_cache() -> &'static Mutex>> { + ARRAY_CACHE.get_or_init(|| Mutex::new(LruCache::new(NonZeroUsize::new(128).unwrap()))) +} + +fn build_array( storage: ReadableWritableListableStorage, path: &str, metadata_json: &str, @@ -21,6 +33,38 @@ fn array_view( Array::new_with_metadata(storage, path, metadata).map_err(value_err) } +/// Construct (or fetch from cache) an Array view from an explicit metadata +/// document, without consulting the store for metadata. When `cache_key` is +/// `Some(root)` the result is memoized on (root, path, metadata_json). +fn array_view( + storage: ReadableWritableListableStorage, + cache_key: Option, + path: &str, + metadata_json: &str, +) -> PyResult> { + if let Some(root) = cache_key { + let key = (root, path.to_string(), metadata_json.to_string()); + if let Some(array) = array_cache().lock().unwrap().get(&key).cloned() { + return Ok(array); + } + let array = Arc::new(build_array(storage, path, metadata_json)?); + array_cache().lock().unwrap().put(key, Arc::clone(&array)); + Ok(array) + } else { + Ok(Arc::new(build_array(storage, path, metadata_json)?)) + } +} + +#[pyfunction] +pub(crate) fn array_cache_len() -> usize { + array_cache().lock().unwrap().len() +} + +#[pyfunction] +pub(crate) fn clear_array_cache() { + array_cache().lock().unwrap().clear(); +} + #[pyfunction] pub(crate) fn retrieve_chunk( py: Python<'_>, @@ -29,9 +73,9 @@ pub(crate) fn retrieve_chunk( metadata_json: String, chunk_coords: Vec, ) -> PyResult> { - let storage = resolve_store(store)?; + let (storage, cache_key) = resolve_store_with_key(store)?; let data = py.detach(move || -> PyResult> { - let array = array_view(storage, &path, &metadata_json)?; + let array = array_view(storage, cache_key, &path, &metadata_json)?; let bytes: ArrayBytes<'static> = array.retrieve_chunk(&chunk_coords).map_err(runtime_err)?; let fixed = bytes.into_fixed().map_err(|_| { @@ -50,9 +94,9 @@ pub(crate) fn retrieve_encoded_chunk( metadata_json: String, chunk_coords: Vec, ) -> PyResult>> { - let storage = resolve_store(store)?; + let (storage, cache_key) = resolve_store_with_key(store)?; let data = py.detach(move || -> PyResult>> { - let array = array_view(storage, &path, &metadata_json)?; + let array = array_view(storage, cache_key, &path, &metadata_json)?; array .retrieve_encoded_chunk(&chunk_coords) .map_err(runtime_err) @@ -69,9 +113,9 @@ pub(crate) fn store_chunk( chunk_coords: Vec, data: Vec, ) -> PyResult<()> { - let storage = resolve_store(store)?; + let (storage, cache_key) = resolve_store_with_key(store)?; py.detach(move || { - let array = array_view(storage, &path, &metadata_json)?; + let array = array_view(storage, cache_key, &path, &metadata_json)?; array .store_chunk(&chunk_coords, ArrayBytes::new_flen(data)) .map_err(runtime_err) @@ -86,9 +130,9 @@ pub(crate) fn erase_chunk( metadata_json: String, chunk_coords: Vec, ) -> PyResult<()> { - let storage = resolve_store(store)?; + let (storage, cache_key) = resolve_store_with_key(store)?; py.detach(move || { - let array = array_view(storage, &path, &metadata_json)?; + let array = array_view(storage, cache_key, &path, &metadata_json)?; array.erase_chunk(&chunk_coords).map_err(runtime_err) }) } @@ -102,9 +146,9 @@ pub(crate) fn retrieve_array_subset( start: Vec, shape: Vec, ) -> PyResult> { - let storage = resolve_store(store)?; + let (storage, cache_key) = resolve_store_with_key(store)?; let data = py.detach(move || -> PyResult> { - let array = array_view(storage, &path, &metadata_json)?; + let array = array_view(storage, cache_key, &path, &metadata_json)?; let subset = ArraySubset::new_with_start_shape(start, shape).map_err(value_err)?; let bytes: ArrayBytes<'static> = array.retrieve_array_subset(&subset).map_err(runtime_err)?; diff --git a/zarrs-bindings/src/lib.rs b/zarrs-bindings/src/lib.rs index aa4552c897..61f947480f 100644 --- a/zarrs-bindings/src/lib.rs +++ b/zarrs-bindings/src/lib.rs @@ -46,5 +46,7 @@ fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_function(wrap_pyfunction!(chunk::store_chunk, m)?)?; m.add_function(wrap_pyfunction!(chunk::erase_chunk, m)?)?; m.add_function(wrap_pyfunction!(chunk::retrieve_array_subset, m)?)?; + m.add_function(wrap_pyfunction!(chunk::array_cache_len, m)?)?; + m.add_function(wrap_pyfunction!(chunk::clear_array_cache, m)?)?; Ok(()) } diff --git a/zarrs-bindings/src/store.rs b/zarrs-bindings/src/store.rs index 3ab8312061..c58de37387 100644 --- a/zarrs-bindings/src/store.rs +++ b/zarrs-bindings/src/store.rs @@ -200,17 +200,26 @@ impl ListableStorageTraits for PyStore { } } -/// Convert the Python-side store representation (`zarr.zarrs._bridge.resolve_store` -/// output) into a zarrs storage handle. -pub(crate) fn resolve_store(obj: &Bound<'_, PyAny>) -> PyResult { +/// Like `resolve_store`, but also returns a cache key for the constructed +/// storage: `Some(root)` for native filesystem stores (which are safe to key an +/// Array cache on), `None` for the generic Python-callback path (uncached). +pub(crate) fn resolve_store_with_key( + obj: &Bound<'_, PyAny>, +) -> PyResult<(ReadableWritableListableStorage, Option)> { if let Ok(config) = obj.cast::() { if let Some(root) = config.get_item("filesystem")? { let root: String = root.extract()?; let store = - FilesystemStore::new(root).map_err(|e| PyValueError::new_err(e.to_string()))?; - return Ok(Arc::new(store)); + FilesystemStore::new(&root).map_err(|e| PyValueError::new_err(e.to_string()))?; + return Ok((Arc::new(store), Some(root))); } return Err(PyValueError::new_err("unrecognized store configuration")); } - Ok(Arc::new(PyStore(obj.clone().unbind()))) + Ok((Arc::new(PyStore(obj.clone().unbind())), None)) +} + +/// Convert the Python-side store representation (`zarr.zarrs._bridge.resolve_store` +/// output) into a zarrs storage handle. +pub(crate) fn resolve_store(obj: &Bound<'_, PyAny>) -> PyResult { + Ok(resolve_store_with_key(obj)?.0) } From 983afbbf34c6436c8dc1ffb02024665cd7ae0b60 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 11:17:50 +0200 Subject: [PATCH 24/75] fix: recover from poisoned Array-cache mutex instead of wedging Replace all four `array_cache().lock().unwrap()` call sites with a `lock_cache()` helper that uses `.unwrap_or_else(|e| e.into_inner())`. If a thread panics while holding the mutex the lock is now recovered (worst case: a stale cache entry) rather than poisoning every subsequent lock call and wedging all array I/O permanently. Co-Authored-By: Claude Fable 5 --- zarrs-bindings/src/chunk.rs | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/zarrs-bindings/src/chunk.rs b/zarrs-bindings/src/chunk.rs index 7d77f10512..556995924b 100644 --- a/zarrs-bindings/src/chunk.rs +++ b/zarrs-bindings/src/chunk.rs @@ -24,6 +24,13 @@ fn array_cache() -> &'static Mutex>> { ARRAY_CACHE.get_or_init(|| Mutex::new(LruCache::new(NonZeroUsize::new(128).unwrap()))) } +/// Acquire the array cache lock, recovering gracefully from a poisoned mutex +/// (e.g. a thread panicked while holding it). The worst case is a stale or +/// partially-updated cache entry — far preferable to wedging all array I/O. +fn lock_cache() -> std::sync::MutexGuard<'static, LruCache>> { + array_cache().lock().unwrap_or_else(|e| e.into_inner()) +} + fn build_array( storage: ReadableWritableListableStorage, path: &str, @@ -44,11 +51,11 @@ fn array_view( ) -> PyResult> { if let Some(root) = cache_key { let key = (root, path.to_string(), metadata_json.to_string()); - if let Some(array) = array_cache().lock().unwrap().get(&key).cloned() { + if let Some(array) = lock_cache().get(&key).cloned() { return Ok(array); } let array = Arc::new(build_array(storage, path, metadata_json)?); - array_cache().lock().unwrap().put(key, Arc::clone(&array)); + lock_cache().put(key, Arc::clone(&array)); Ok(array) } else { Ok(Arc::new(build_array(storage, path, metadata_json)?)) @@ -57,12 +64,12 @@ fn array_view( #[pyfunction] pub(crate) fn array_cache_len() -> usize { - array_cache().lock().unwrap().len() + lock_cache().len() } #[pyfunction] pub(crate) fn clear_array_cache() { - array_cache().lock().unwrap().clear(); + lock_cache().clear(); } #[pyfunction] From 0087a7e44ed6537c40fc6e8d1cc30a71dad5f734 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 11:19:04 +0200 Subject: [PATCH 25/75] docs: record Array construction cache in design spec Co-Authored-By: Claude Fable 5 --- .../2026-06-11-zarrs-functional-api-design.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md index bfafd59aae..0c7deb8eac 100644 --- a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md +++ b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md @@ -136,6 +136,30 @@ function calls a blocking Rust entry point via `asyncio.to_thread`; the Rust side releases the GIL during I/O and compute (reacquiring it only inside `PyStore` callbacks). zarrs's experimental async feature is not used. +## Array construction cache + +`Array::new_with_metadata` (serde-parsing the metadata document and building the +codec chain) is the dominant per-call cost on the native path — measured at +~20µs for a bytes-only array up to ~80µs for sharded+blosc, against single-digit +µs of actual chunk I/O on a warm filesystem. To amortize it across the common +"open one array, then do many chunk operations" pattern, the chunk/region +routines memoize the constructed `Array` in a process-global LRU cache +(capacity 128) keyed on `(filesystem root, node path, metadata JSON)`. + +This is safe because a zarrs `Array` caches no chunk data — it is metadata plus +codec chain plus a storage handle — so every read/write still goes through to +the store, and a correctly-keyed hit is behaviorally identical to a fresh build. +The key must include all three components: the same document at a different path +or store is a different array. Only native filesystem stores are cached; the +generic `PyStore` callback path has no stable cross-call identity to key on and +is left uncached (a future change may cache it if a store can supply a stable +value-based token). No invalidation hook is needed: delete/overwrite with +different metadata yields a different key, and an entry for a deleted-and-rebuilt +array with identical metadata stays valid because reads go through to the store. +A poisoned cache mutex is recovered rather than propagated, so the cache can +never wedge array I/O. Measured win: 14–20% faster per repeated call on a local +store, free on every hit. + ## Error handling The binding layer raises a small set of typed exceptions defined in one place: From cb87cae91dc3e8853b6283b43aee864e09643981 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 12:05:03 +0200 Subject: [PATCH 26/75] docs: design spec for backend-agnostic CRUD layer Co-Authored-By: Claude Fable 5 --- ...6-06-15-crud-backend-abstraction-design.md | 177 ++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md diff --git a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md new file mode 100644 index 0000000000..02ab480913 --- /dev/null +++ b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md @@ -0,0 +1,177 @@ +# Backend-agnostic CRUD layer for zarr-python + +Date: 2026-06-15 +Status: approved +Branch: `zarrs-bindings` + +## Goal + +Turn the low-level functional CRUD API (introduced as `zarr.zarrs` earlier on +this branch) into a backend-agnostic layer, with the Rust zarrs bindings as one +of several interchangeable implementations. Define the CRUD contract abstractly, +provide a pure-Python reference backend (no Rust required), and make the zarrs +bindings conform to the same contract. + +This validates the abstraction by having two real backends agree with each other +and with zarr-python, and it gives users a no-Rust fallback. + +Non-goals for this change (deliberately deferred): + +- Wiring the CRUD layer under zarr-python's own `Array`/`Group` classes. +- Entrypoint-based backend discovery (this change uses explicit import-time + registration). +- Region/selection features beyond what already exists (`encode_region`, + chunk-subset `selection` on `decode_chunk` remain future work, unchanged). + +## Background + +The current `zarr.zarrs._api` is a flat module of 13 async functions that +delegate to the `_zarrs_bindings` Rust extension. It already separates two +concerns that this design formalizes into a hard boundary: + +- **Backend-neutral glue:** `_normalize_selection`, `_array_shape`, + `_chunk_dtype_and_shape`, numpy assembly (`np.frombuffer`/reshape/strided + views), native-dtype coercion, options handling, error translation. +- **Genuinely zarrs-specific work:** producing/consuming raw chunk bytes, + reading array subsets as bytes, writing metadata documents — all via + `_zarrs_bindings` and the `_bridge.StoreShim`/`resolve_store` plumbing. + +The public surface (`zarr.zarrs.decode_region`, etc.) is unreleased on this +branch, so it can move without backward-compatibility constraints. + +zarr-python already contains everything a pure-Python backend needs: +`BatchedCodecPipeline` (`src/zarr/core/codec_pipeline.py`), `BasicIndexer` +(`src/zarr/core/indexing.py`), `save_metadata` (`src/zarr/core/metadata/io.py`), +metadata parsing (`ArrayV3Metadata.from_dict` / `ArrayV2Metadata.from_dict`), +and chunk-key encoding (`src/zarr/core/chunk_key_encodings.py`). + +## Architecture + +Two packages with a hard boundary. + +### `zarr.crud` (new, backend-neutral) + +- `_backend.py` — the `CrudBackend` `Protocol` (the narrow byte/metadata + contract below) plus the canonical exceptions. +- `_api.py` — the shared async facade: the 13 public functions moved out of + `zarr.zarrs`, holding all backend-neutral logic. Each function resolves a + backend (from the `backend` argument or the registry default) and calls its + byte/metadata methods, then does selection normalization, dtype handling, and + numpy assembly. +- `_reference.py` — `ReferenceBackend`, pure Python, wrapping zarr-python's own + codec/indexing/metadata machinery. Always importable; the default backend. +- `_registry.py` — `register_backend(name, backend)`, `get_backend(name)`, and + the config-driven default resolution. +- `__init__.py` — re-exports the facade functions, `CrudBackend`, + `ZarrsOptions`, the exceptions, and `register_backend`. + +### `zarr.zarrs` (shrinks to the zarrs provider) + +- `_backend.py` — `ZarrsBackend`, implementing `CrudBackend` by wrapping + `_zarrs_bindings`. Owns the zarrs-isms that move out of the facade: + `json.dumps` of the metadata dict, the `/`-prefixed zarrs node-path form + (formerly `_node_path`), `_bridge.resolve_store`, and translation of + `_zarrs_bindings` exceptions into the `zarr.crud` canonical exceptions. +- `_bridge.py` — unchanged (`StoreShim`, `resolve_store`). +- the Rust crate `zarrs-bindings/` and the construction cache — unchanged. +- registers itself as backend `"zarrs"` at import time. + +## The `CrudBackend` contract + +Narrow, byte/metadata level. Methods pass neutral types — the metadata document +as a `dict`, the zarr `Store`, and plain zarr paths (`""`, `"foo/bar"`) — and +return raw bytes / JSON-as-dict / `None`. Each backend serializes and bridges as +it needs. + +```python +class CrudBackend(Protocol): + async def create_array(self, store, path, metadata, *, overwrite: bool) -> None: ... + async def create_group(self, store, path, metadata, *, overwrite: bool) -> None: ... + async def read_metadata(self, store, path) -> dict[str, JSON]: ... + async def delete_node(self, store, path) -> None: ... + async def list_children(self, store, path) -> list[tuple[str, dict[str, JSON]]]: ... + async def retrieve_chunk(self, store, path, metadata, coords) -> bytes: ... + async def retrieve_encoded_chunk(self, store, path, metadata, coords) -> bytes | None: ... + async def retrieve_subset(self, store, path, metadata, start, shape) -> bytes: ... + async def store_chunk(self, store, path, metadata, coords, data: bytes) -> None: ... + async def erase_chunk(self, store, path, metadata, coords) -> None: ... +``` + +Byte conventions: `retrieve_chunk`/`retrieve_subset` return C-contiguous raw +bytes in the array's native byte order for the requested chunk / step-1 bounding +box; `store_chunk` takes the same. `retrieve_encoded_chunk` returns the raw +stored (still-encoded) chunk bytes or `None` if absent. `read_metadata`/ +`list_children` return parsed JSON documents as dicts. + +## Facade / backend split + +What stays in the `zarr.crud` facade (written once, backend-neutral): + +- selection normalization (`_normalize_selection`), shape/dtype resolution + (`_array_shape`, `_chunk_dtype_and_shape`), native-dtype coercion; +- numpy assembly: `np.frombuffer(...).reshape(...)` and the strided/reversed/ + integer-axis post-index views; read-only result guarantee; +- the empty-selection short circuit (no backend call); +- `ZarrsOptions` acceptance (still a placeholder) and backend resolution. + +What moves into each backend: + +- `ZarrsBackend`: `json.dumps`, the `/`-prefixed node-path form, + `resolve_store`, calling `_zarrs_bindings`, exception translation. +- `ReferenceBackend`: `ArrayV3Metadata.from_dict`/`ArrayV2Metadata.from_dict`, + building a `BatchedCodecPipeline` and `ChunkGrid`/`BasicIndexer`, assembling + `batch_info` and calling `codec_pipeline.read`/`write`, `save_metadata`, + `store.delete_dir`, and `list_dir` + per-child metadata reads. + +## Backend selection + +- A registry in `zarr.crud._registry`: `register_backend(name, backend)`, + `get_backend(name) -> CrudBackend`. +- A `zarr.config` key `crud.backend`, default `"reference"`. The pure-Python + backend always works and is predictable; `"zarrs"` opts into the accelerator + and is registered when `zarr.zarrs` is imported. +- Every facade function accepts `backend: CrudBackend | str | None = None`. + `None` → registry default; a string → registry lookup; an instance → used + directly. This enables side-by-side testing of backends. + +## Error handling + +`zarr.crud` defines the canonical exceptions: reuse +`zarr.errors.NodeNotFoundError`, and keep a `NodeExistsError` (exposed as +`zarr.crud.NodeExistsError`). Each backend raises these directly: + +- `ReferenceBackend` raises them at the point of detection. +- `ZarrsBackend` translates `_zarrs_bindings.NodeExistsError` / + `_zarrs_bindings.NodeNotFoundError` into the canonical types. + +The facade therefore no longer needs the `_translate_errors` shim. Phase-1 +fidelity limits (store-callback exceptions flattened to `RuntimeError` across the +Rust boundary) are unchanged for the zarrs backend; the reference backend +surfaces native exceptions directly. + +## Testing + +- Shared differential suite moves to `tests/crud/`, parametrized over + `backend ∈ {reference, zarrs}` × `store ∈ {memory, local}`. Each test writes + with zarr-python and reads through the facade (and vice versa), so the two + backends are checked against zarr-python *and*, transitively, against each + other. The zarrs-parametrized cases skip when `_zarrs_bindings` is not + installed (xdist-safe module-level `importorskip` in a zarrs-only conftest + helper, or a skip marker on the zarrs param). +- Zarrs-only tests stay in `tests/zarrs/`: the construction cache + (`test_cache.py`) and the store bridge (`test_bridge.py`). +- A focused `tests/crud/test_registry.py`: default resolution, `register_backend`, + string vs instance `backend=` override. +- `uv run --group zarrs pytest tests/crud tests/zarrs` is the full local check; + `uv run pytest tests/crud` (no zarrs group) must pass with the reference + backend alone and skip the zarrs params. + +## Migration notes + +- Move the 13 functions and the neutral helpers from `zarr.zarrs._api` into + `zarr.crud._api`; delete `zarr.zarrs._api`. No aliases in `zarr.zarrs`. +- `zarr.zarrs.__init__` exports only what is needed to register and identify the + zarrs backend (`ZarrsBackend`, and re-registers `"zarrs"` on import). +- The changelog fragment is updated to describe `zarr.crud` as the public CRUD + surface with pluggable backends, and `zarr.zarrs` as the zarrs backend. +- The CI job continues to build the crate and now runs `tests/crud tests/zarrs`. From 5ae3bb21a395387695cae9d7c9aed6434f56275d Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 12:13:06 +0200 Subject: [PATCH 27/75] docs: consistent CRUD verb set (create/read/write/delete/list) Co-Authored-By: Claude Fable 5 --- ...6-06-15-crud-backend-abstraction-design.md | 55 +++++++++++++++---- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md index 02ab480913..2b4946a812 100644 --- a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md +++ b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md @@ -21,7 +21,7 @@ Non-goals for this change (deliberately deferred): - Entrypoint-based backend discovery (this change uses explicit import-time registration). - Region/selection features beyond what already exists (`encode_region`, - chunk-subset `selection` on `decode_chunk` remain future work, unchanged). + chunk-subset `selection` on `read_chunk` remain future work, unchanged). ## Background @@ -88,21 +88,51 @@ class CrudBackend(Protocol): async def create_array(self, store, path, metadata, *, overwrite: bool) -> None: ... async def create_group(self, store, path, metadata, *, overwrite: bool) -> None: ... async def read_metadata(self, store, path) -> dict[str, JSON]: ... + async def read_chunk(self, store, path, metadata, coords) -> bytes: ... + async def read_encoded_chunk(self, store, path, metadata, coords) -> bytes | None: ... + async def read_subset(self, store, path, metadata, start, shape) -> bytes: ... + async def write_chunk(self, store, path, metadata, coords, data: bytes) -> None: ... + async def delete_chunk(self, store, path, metadata, coords) -> None: ... async def delete_node(self, store, path) -> None: ... async def list_children(self, store, path) -> list[tuple[str, dict[str, JSON]]]: ... - async def retrieve_chunk(self, store, path, metadata, coords) -> bytes: ... - async def retrieve_encoded_chunk(self, store, path, metadata, coords) -> bytes | None: ... - async def retrieve_subset(self, store, path, metadata, start, shape) -> bytes: ... - async def store_chunk(self, store, path, metadata, coords, data: bytes) -> None: ... - async def erase_chunk(self, store, path, metadata, coords) -> None: ... ``` -Byte conventions: `retrieve_chunk`/`retrieve_subset` return C-contiguous raw -bytes in the array's native byte order for the requested chunk / step-1 bounding -box; `store_chunk` takes the same. `retrieve_encoded_chunk` returns the raw -stored (still-encoded) chunk bytes or `None` if absent. `read_metadata`/ +Byte conventions: `read_chunk`/`read_subset` return C-contiguous raw bytes in the +array's native byte order for the requested chunk / step-1 bounding box; +`write_chunk` takes the same. `read_encoded_chunk` returns the raw stored +(still-encoded) chunk bytes or `None` if absent. `read_metadata`/ `list_children` return parsed JSON documents as dicts. +## Method naming + +Both the public facade and the backend contract use a single, consistent verb +set: **create / read / write / delete / list**. No `decode`/`encode`/`retrieve`/ +`store`/`erase` synonyms. + +Public facade (`zarr.crud`): + +| Function | Verb | Notes | +|---|---|---| +| `create_new_group` / `create_overwrite_group` | create | node lifecycle | +| `create_new_array` / `create_overwrite_array` | create | node lifecycle | +| `read_metadata` | read | array or group document | +| `read_chunk` | read | decoded chunk → `ndarray` | +| `read_encoded_chunk` | read | raw stored bytes, no decode | +| `read_region` | read | numpy basic-indexing selection → `ndarray` | +| `write_chunk` | write | encode + store a chunk | +| `delete_chunk` | delete | remove one chunk | +| `delete_node` | delete | remove a node + descendants | +| `list_children` | list | direct children of a group | + +Facade → backend mapping for the byte-level methods: `read_chunk` → +`backend.read_chunk`, `read_encoded_chunk` → `backend.read_encoded_chunk`, +`read_region` → `backend.read_subset` (the facade normalizes the selection to a +step-1 bounding box `(start, shape)`), `write_chunk` → `backend.write_chunk`, +`delete_chunk` → `backend.delete_chunk`. The two distinct names `read_region` +(selection-based, public) and `read_subset` (bounding-box bytes, backend) are +intentional: they have different signatures and the facade is the adapter +between them. + ## Facade / backend split What stays in the `zarr.crud` facade (written once, backend-neutral): @@ -170,6 +200,11 @@ surfaces native exceptions directly. - Move the 13 functions and the neutral helpers from `zarr.zarrs._api` into `zarr.crud._api`; delete `zarr.zarrs._api`. No aliases in `zarr.zarrs`. +- Rename to the consistent verb set in the move (no compatibility aliases, since + the surface is unreleased): `decode_chunk` → `read_chunk`, `decode_region` → + `read_region`, `encode_chunk` → `write_chunk`, `erase_chunk` → `delete_chunk`. + `read_metadata`, `read_encoded_chunk`, `delete_node`, `list_children`, and the + `create_*` functions keep their names. - `zarr.zarrs.__init__` exports only what is needed to register and identify the zarrs backend (`ZarrsBackend`, and re-registers `"zarrs"` on import). - The changelog fragment is updated to describe `zarr.crud` as the public CRUD From c7af3053af347857118fb4cf3ceb0c2962c97a86 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 12:19:13 +0200 Subject: [PATCH 28/75] docs: simplify CRUD reads to two non-overlapping addressing axes read_chunk stays parameter-free (chunk-grid addressed, whole chunk); all selection-based reads route through read_region/read_subset (array-coordinate, spans chunks). Drops the deferred chunk-subset selection parameter. Co-Authored-By: Claude Fable 5 --- ...6-06-15-crud-backend-abstraction-design.md | 26 +++++++++++++++++-- 1 file changed, 24 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md index 2b4946a812..349c77fa73 100644 --- a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md +++ b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md @@ -20,8 +20,7 @@ Non-goals for this change (deliberately deferred): - Wiring the CRUD layer under zarr-python's own `Array`/`Group` classes. - Entrypoint-based backend discovery (this change uses explicit import-time registration). -- Region/selection features beyond what already exists (`encode_region`, - chunk-subset `selection` on `read_chunk` remain future work, unchanged). +- A write-side region operation (`write_region`) remains future work. ## Background @@ -103,6 +102,29 @@ array's native byte order for the requested chunk / step-1 bounding box; (still-encoded) chunk bytes or `None` if absent. `read_metadata`/ `list_children` return parsed JSON documents as dicts. +### Two read-addressing axes (no overlap) + +Reads are addressed in one of two coordinate spaces, and the two never overlap: + +- **Chunk-grid coordinates** — `read_chunk(coords)` / `read_encoded_chunk(coords)` + return a whole chunk addressed by its grid position. `read_chunk` returns the + *full* chunk shape, including the fill-padded overhang of edge chunks; + `read_encoded_chunk` returns the raw stored bytes. These pair with + `write_chunk` / `delete_chunk`, which are also chunk-grid-addressed. +- **Array-element coordinates** — `read_subset(start, shape)` returns an + arbitrary box in array space, which generally spans multiple chunks and is + clipped to the array bounds. The facade's `read_region(selection)` normalizes + a numpy selection to a step-1 bounding box and calls it. + +`read_chunk` takes no `selection` parameter. A sub-region *within* a single +chunk is simply a `read_region` whose bounding box lies inside one chunk; the +backend already decodes only the overlapping chunk(s) (sharding-aware in the +zarrs backend), so a chunk-relative partial-read needs no separate API. The +`Store.get(key, byte_range=)` analogue is therefore `read_region` over a +single-chunk box, not a parameter on `read_chunk`; `read_subset` itself has no +single-`get` analogue — it is closer to "`get_partial_values` across many keys, +stitched into one array." + ## Method naming Both the public facade and the backend contract use a single, consistent verb From 3740888d2ccf97c369aea5ccf4710cd79118015a Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 12:29:15 +0200 Subject: [PATCH 29/75] docs: make read_encoded_chunk a facade store.get helper, not a backend method Co-Authored-By: Claude Fable 5 --- ...6-06-15-crud-backend-abstraction-design.md | 57 +++++++++++++------ 1 file changed, 41 insertions(+), 16 deletions(-) diff --git a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md index 349c77fa73..bf8043513b 100644 --- a/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md +++ b/docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md @@ -88,7 +88,6 @@ class CrudBackend(Protocol): async def create_group(self, store, path, metadata, *, overwrite: bool) -> None: ... async def read_metadata(self, store, path) -> dict[str, JSON]: ... async def read_chunk(self, store, path, metadata, coords) -> bytes: ... - async def read_encoded_chunk(self, store, path, metadata, coords) -> bytes | None: ... async def read_subset(self, store, path, metadata, start, shape) -> bytes: ... async def write_chunk(self, store, path, metadata, coords, data: bytes) -> None: ... async def delete_chunk(self, store, path, metadata, coords) -> None: ... @@ -96,21 +95,24 @@ class CrudBackend(Protocol): async def list_children(self, store, path) -> list[tuple[str, dict[str, JSON]]]: ... ``` +Nine methods. `read_encoded_chunk` is deliberately **not** a backend method — +see below; it is a backend-independent facade helper over `store.get`. + Byte conventions: `read_chunk`/`read_subset` return C-contiguous raw bytes in the array's native byte order for the requested chunk / step-1 bounding box; -`write_chunk` takes the same. `read_encoded_chunk` returns the raw stored -(still-encoded) chunk bytes or `None` if absent. `read_metadata`/ -`list_children` return parsed JSON documents as dicts. +`write_chunk` takes the same. `read_metadata`/`list_children` return parsed JSON +documents as dicts. ### Two read-addressing axes (no overlap) Reads are addressed in one of two coordinate spaces, and the two never overlap: - **Chunk-grid coordinates** — `read_chunk(coords)` / `read_encoded_chunk(coords)` - return a whole chunk addressed by its grid position. `read_chunk` returns the - *full* chunk shape, including the fill-padded overhang of edge chunks; - `read_encoded_chunk` returns the raw stored bytes. These pair with - `write_chunk` / `delete_chunk`, which are also chunk-grid-addressed. + return a whole chunk addressed by its grid position. `read_chunk` (a backend + method) decodes and returns the *full* chunk shape, including the fill-padded + overhang of edge chunks; `read_encoded_chunk` (a facade helper, not a backend + method) returns the raw stored bytes or `None`. These pair with `write_chunk` / + `delete_chunk`, which are chunk-grid-addressed backend methods. - **Array-element coordinates** — `read_subset(start, shape)` returns an arbitrary box in array space, which generally spans multiple chunks and is clipped to the array bounds. The facade's `read_region(selection)` normalizes @@ -125,6 +127,27 @@ single-chunk box, not a parameter on `read_chunk`; `read_subset` itself has no single-`get` analogue — it is closer to "`get_partial_values` across many keys, stitched into one array." +### `read_encoded_chunk` is facade-level, not a backend method + +Reading a chunk's raw stored bytes is just `store.get(chunk_key)`, and the chunk +key is computable from the metadata document alone via zarr-python's +`chunk_key_encoding` — no decoding, no codec pipeline, nothing backend-specific. +Both backends would implement it identically. So the facade implements +`read_encoded_chunk(store, path, metadata, coords)` directly as: encode the chunk +key from the metadata, `store.get` it, return the bytes or `None`. It works the +same regardless of which backend (or none) is selected, which is correct since it +is pure store I/O. Under sharding the chunk key holds the whole shard blob, and +this returns exactly that raw object. + +This raw read can also be *expressed* through `read_chunk` by supplying a view +metadata document (`data_type: uint8` + a single `bytes` codec, identity decode) +— a nice demonstration that the read-only-view mechanism is general — but that +route requires knowing the encoded byte length up front to set the chunk shape (a +`store.getsize` round-trip) and would synthesize a fill-valued array for a missing +chunk instead of returning `None`. So `store.get` is the correct implementation +for fetching stored bytes; the view trick is the general tool for *reinterpreting* +decoded data under a different dtype/shape, which `read_chunk` already supports. + ## Method naming Both the public facade and the backend contract use a single, consistent verb @@ -139,7 +162,7 @@ Public facade (`zarr.crud`): | `create_new_array` / `create_overwrite_array` | create | node lifecycle | | `read_metadata` | read | array or group document | | `read_chunk` | read | decoded chunk → `ndarray` | -| `read_encoded_chunk` | read | raw stored bytes, no decode | +| `read_encoded_chunk` | read | raw stored bytes, no decode (facade-only, `store.get`) | | `read_region` | read | numpy basic-indexing selection → `ndarray` | | `write_chunk` | write | encode + store a chunk | | `delete_chunk` | delete | remove one chunk | @@ -147,13 +170,13 @@ Public facade (`zarr.crud`): | `list_children` | list | direct children of a group | Facade → backend mapping for the byte-level methods: `read_chunk` → -`backend.read_chunk`, `read_encoded_chunk` → `backend.read_encoded_chunk`, -`read_region` → `backend.read_subset` (the facade normalizes the selection to a -step-1 bounding box `(start, shape)`), `write_chunk` → `backend.write_chunk`, -`delete_chunk` → `backend.delete_chunk`. The two distinct names `read_region` -(selection-based, public) and `read_subset` (bounding-box bytes, backend) are -intentional: they have different signatures and the facade is the adapter -between them. +`backend.read_chunk`, `read_region` → `backend.read_subset` (the facade +normalizes the selection to a step-1 bounding box `(start, shape)`), +`write_chunk` → `backend.write_chunk`, `delete_chunk` → `backend.delete_chunk`. +`read_encoded_chunk` maps to no backend method — the facade serves it from +`store.get`. The two distinct names `read_region` (selection-based, public) and +`read_subset` (bounding-box bytes, backend) are intentional: they have different +signatures and the facade is the adapter between them. ## Facade / backend split @@ -164,6 +187,8 @@ What stays in the `zarr.crud` facade (written once, backend-neutral): - numpy assembly: `np.frombuffer(...).reshape(...)` and the strided/reversed/ integer-axis post-index views; read-only result guarantee; - the empty-selection short circuit (no backend call); +- `read_encoded_chunk`: encode the chunk key from the metadata and `store.get` + it (no backend involved); - `ZarrsOptions` acceptance (still a placeholder) and backend resolution. What moves into each backend: From a9cdae6fb5103fde95ea1a73da77a090ab54c4ec Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 13:38:37 +0200 Subject: [PATCH 30/75] docs: implementation plan for backend-agnostic CRUD layer Co-Authored-By: Claude Fable 5 --- .../2026-06-15-crud-backend-abstraction.md | 1698 +++++++++++++++++ 1 file changed, 1698 insertions(+) create mode 100644 docs/superpowers/plans/2026-06-15-crud-backend-abstraction.md diff --git a/docs/superpowers/plans/2026-06-15-crud-backend-abstraction.md b/docs/superpowers/plans/2026-06-15-crud-backend-abstraction.md new file mode 100644 index 0000000000..985f325990 --- /dev/null +++ b/docs/superpowers/plans/2026-06-15-crud-backend-abstraction.md @@ -0,0 +1,1698 @@ +# Backend-agnostic CRUD layer Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Turn the low-level functional CRUD API into a backend-agnostic `zarr.crud` package with a pure-Python reference backend and the existing zarrs bindings as a second, interchangeable backend. + +**Architecture:** A narrow async `CrudBackend` protocol (byte/metadata level) plus a shared `zarr.crud` facade that holds all backend-neutral logic (selection normalization, numpy assembly, dtype handling, `read_encoded_chunk` via `store.get`). Two backends conform: `ReferenceBackend` (pure Python, wraps zarr-python's own codec pipeline / indexer / metadata machinery) and `ZarrsBackend` (wraps `_zarrs_bindings`). A registry + `zarr.config` key `crud.backend` (default `"reference"`) selects one; every facade function also takes `backend=`. + +**Tech Stack:** Python 3.12+, numpy, zarr-python internals (`BatchedCodecPipeline`, `AsyncArray`, `save_metadata`, `ArrayConfig`/`ArraySpec`, chunk-key encoding), the existing `_zarrs_bindings` Rust extension (unchanged — no Rust build needed). + +Spec: `docs/superpowers/specs/2026-06-15-crud-backend-abstraction-design.md`. + +--- + +## Environment notes (read first) + +- **Run python/pytest/mypy via `uv run`.** The zarrs backend needs the extension: `uv run --group zarrs pytest ...`. The reference backend works under plain `uv run pytest ...`. +- The Claude Code bash sandbox is broken on this host (`bwrap: loopback` error). Run commands with the sandbox **disabled**. +- **No Rust changes in this plan.** The `_zarrs_bindings` pyfunctions keep their existing names (`retrieve_chunk`, `store_chunk`, `erase_chunk`, `retrieve_array_subset`, `retrieve_encoded_chunk`, `create_array`, `create_group`, `read_metadata`, `delete_node`, `list_children`); `ZarrsBackend` adapts them to the contract's verb names. No `cargo` build or `uv sync --reinstall` is required, but the `zarrs` group must already be installed (`uv sync --group zarrs`) to run the zarrs-parametrized tests. +- Pre-commit hooks (ruff strict, mypy strict over `src`+`tests`, codespell) run on `git commit`. If a hook rewrites a file, `git add` and commit again. +- Docstrings use markdown (single backticks), not RST. +- pytest is configured with `asyncio_mode = "auto"` — async tests/fixtures need no decorator. + +## File structure + +``` +src/zarr/crud/ + __init__.py # public exports; registers the reference backend at import + _backend.py # CrudBackend Protocol + NodeExistsError + _registry.py # register_backend / get_backend + config default resolution + _reference.py # ReferenceBackend (pure Python) + _api.py # shared async facade (the public functions) + neutral helpers +src/zarr/zarrs/ + __init__.py # SHRINKS: version + register ZarrsBackend; no _api re-exports + _backend.py # ZarrsBackend (wraps _zarrs_bindings) — NEW + _bridge.py # unchanged + _api.py # DELETED +src/zarr/core/config.py # add "crud": {"backend": "reference"} +tests/crud/ + __init__.py + conftest.py # store fixture, backend fixture (reference+zarrs), metadata helpers + test_registry.py # registry + default + override + test_reference_backend.py # direct reference-backend smoke tests + test_crud.py # full differential suite, parametrized over backend x store +tests/zarrs/ + __init__.py # unchanged + conftest.py # unchanged (still used by test_bridge/test_cache) + test_bridge.py # unchanged + test_cache.py # imports updated to zarr.crud read_chunk/write_chunk, backend="zarrs" + test_node.py # DELETED (covered by tests/crud/test_crud.py) + test_chunk.py # DELETED (covered by tests/crud/test_crud.py) + test_api.py # DELETED (replaced by tests/crud import coverage) +changes/+zarrs-bindings.feature.md # reworded for zarr.crud +.github/workflows/zarrs.yml # run tests/crud tests/zarrs +``` + +--- + +### Task 1: `zarr.crud` skeleton — protocol, exceptions, registry, config + +**Files:** +- Create: `src/zarr/crud/__init__.py` +- Create: `src/zarr/crud/_backend.py` +- Create: `src/zarr/crud/_registry.py` +- Modify: `src/zarr/core/config.py` +- Create: `tests/crud/__init__.py` (empty) +- Test: `tests/crud/test_registry.py` + +- [ ] **Step 1: Write the failing test** — `tests/crud/test_registry.py` + +```python +from __future__ import annotations + +import pytest + +from zarr.crud import CrudBackend, NodeExistsError, get_backend, register_backend + + +def test_node_exists_error_is_value_error() -> None: + assert issubclass(NodeExistsError, ValueError) + + +def test_default_backend_is_reference() -> None: + # the reference backend is registered at import and is the configured default + be = get_backend() + assert be is get_backend("reference") + + +def test_get_unknown_backend_raises() -> None: + with pytest.raises(KeyError, match="no CRUD backend"): + get_backend("does-not-exist") + + +def test_register_and_resolve_instance() -> None: + class Dummy: + pass + + dummy = Dummy() + register_backend("dummy-test", dummy) # type: ignore[arg-type] + try: + assert get_backend("dummy-test") is dummy + finally: + from zarr.crud import _registry + + _registry._BACKENDS.pop("dummy-test", None) + + +def test_protocol_is_runtime_checkable() -> None: + # ReferenceBackend (registered as "reference") structurally satisfies the protocol + assert isinstance(get_backend("reference"), CrudBackend) +``` + +- [ ] **Step 2: Run it to verify failure** + +Run: `uv run pytest tests/crud/test_registry.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'zarr.crud'` + +- [ ] **Step 3: Create `tests/crud/__init__.py`** (empty file) + +- [ ] **Step 4: Create `src/zarr/crud/_backend.py`** + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol, runtime_checkable + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + from zarr.abc.store import Store + from zarr.core.common import JSON + + +class NodeExistsError(ValueError): + """Raised when a node already exists at a path and overwrite was not requested.""" + + +@runtime_checkable +class CrudBackend(Protocol): + """The byte/metadata-level contract a CRUD backend must implement. + + Methods take neutral types: the metadata document as a `dict`, a zarr + `Store`, and plain zarr paths (`""`, `"foo/bar"`). They return raw bytes, + parsed JSON documents, or `None`. The shared `zarr.crud` facade builds the + numpy- and selection-level API on top of these. + + `create_*` raise `zarr.crud.NodeExistsError` when a node exists and + `overwrite` is false. `read_metadata`/`delete_node`/`list_children` raise + `zarr.errors.NodeNotFoundError` when the target is missing. + """ + + async def create_array( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: ... + + async def create_group( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: ... + + async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: ... + + async def read_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> bytes: ... + + async def read_subset( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + start: Sequence[int], + shape: Sequence[int], + ) -> bytes: ... + + async def write_chunk( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + coords: tuple[int, ...], + data: bytes, + ) -> None: ... + + async def delete_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> None: ... + + async def delete_node(self, store: Store, path: str) -> None: ... + + async def list_children( + self, store: Store, path: str + ) -> list[tuple[str, dict[str, JSON]]]: ... +``` + +- [ ] **Step 5: Create `src/zarr/crud/_registry.py`** + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr.core.config import config + +if TYPE_CHECKING: + from zarr.crud._backend import CrudBackend + +_BACKENDS: dict[str, CrudBackend] = {} + + +def register_backend(name: str, backend: CrudBackend) -> None: + """Register a CRUD backend instance under `name`.""" + _BACKENDS[name] = backend + + +def get_backend(name: str | None = None) -> CrudBackend: + """Resolve a backend by name, or the configured default when `name` is None. + + Selecting `"zarrs"` imports `zarr.zarrs` if needed so it can self-register. + """ + if name is None: + name = config.get("crud.backend") + if name not in _BACKENDS and name == "zarrs": + import zarr.zarrs # noqa: F401 (import registers the zarrs backend) + if name not in _BACKENDS: + raise KeyError( + f"no CRUD backend registered as {name!r}; registered: {sorted(_BACKENDS)}" + ) + return _BACKENDS[name] +``` + +- [ ] **Step 6: Create `src/zarr/crud/__init__.py`** (reference backend is added in Task 2; for now register nothing) + +```python +""" +Backend-agnostic low-level functional CRUD API for zarr hierarchies. + +The public functions delegate byte- and metadata-level work to a `CrudBackend`. +Two backends ship: a pure-Python reference backend (the default) and a +zarrs-accelerated backend (`zarr.zarrs`, requires the `zarrs-bindings` +extension). Select one with the `crud.backend` config key or a per-call +`backend=` argument. + +Array routines take an explicit metadata document (a `dict` matching the +`zarr.json` / `.zarray` document) rather than reading it from the store, which +makes read-only and virtual views possible. +""" + +from zarr.crud._backend import CrudBackend, NodeExistsError +from zarr.crud._registry import get_backend, register_backend + +__all__ = [ + "CrudBackend", + "NodeExistsError", + "get_backend", + "register_backend", +] +``` + +- [ ] **Step 7: Add the config default** — `src/zarr/core/config.py` + +Find the defaults mapping passed to the `Config(...)` constructor (it contains the `"codec_pipeline"` key). Add a sibling entry: + +```python + "crud": {"backend": "reference"}, +``` + +Run to confirm it loads: `uv run python -c "from zarr.core.config import config; print(config.get('crud.backend'))"` +Expected: `reference` + +- [ ] **Step 8: Run the test (note: `test_default_backend_is_reference` and the protocol test still fail — reference backend arrives in Task 2)** + +Run: `uv run pytest tests/crud/test_registry.py -v` +Expected: `test_node_exists_error_is_value_error`, `test_get_unknown_backend_raises`, `test_register_and_resolve_instance` PASS; `test_default_backend_is_reference` and `test_protocol_is_runtime_checkable` FAIL (KeyError: no backend `reference`). That is expected at this task boundary; they pass after Task 2. + +- [ ] **Step 9: Commit** + +```bash +git add src/zarr/crud/_backend.py src/zarr/crud/_registry.py src/zarr/crud/__init__.py src/zarr/core/config.py tests/crud/__init__.py tests/crud/test_registry.py +git commit -m "feat: zarr.crud skeleton — CrudBackend protocol, registry, config" +``` + +End every commit body in this plan with: +``` +Co-Authored-By: Claude Fable 5 +``` + +--- + +### Task 2: `ReferenceBackend` (pure Python) + +**Files:** +- Create: `src/zarr/crud/_reference.py` +- Modify: `src/zarr/crud/__init__.py` +- Test: `tests/crud/test_reference_backend.py` + +All snippets below are verified against the installed zarr-python. + +- [ ] **Step 1: Write the failing test** — `tests/crud/test_reference_backend.py` + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING + +import numpy as np + +import zarr +from zarr.crud import NodeExistsError, get_backend +from zarr.errors import NodeNotFoundError +from zarr.storage import MemoryStore + +if TYPE_CHECKING: + pass + +import pytest + + +def _array_meta() -> dict: + arr = zarr.create_array(store=MemoryStore(), shape=(8, 8), chunks=(4, 4), dtype="uint16") + return dict(arr.metadata.to_dict()) + + +async def test_reference_round_trip_chunk() -> None: + be = get_backend("reference") + store = MemoryStore() + meta = _array_meta() + await be.create_array(store, "a", meta, overwrite=False) + value = np.arange(16, dtype="uint16").reshape(4, 4) + await be.write_chunk(store, "a", meta, (0, 1), value.tobytes()) + raw = await be.read_chunk(store, "a", meta, (0, 1)) + np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(4, 4), value) + + +async def test_reference_read_subset_spans_chunks() -> None: + be = get_backend("reference") + store = MemoryStore() + arr = zarr.create_array(store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16") + data = np.arange(64, dtype="uint16").reshape(8, 8) + arr[:, :] = data + meta = dict(arr.metadata.to_dict()) + raw = await be.read_subset(store, "a", meta, (2, 1), (5, 4)) + np.testing.assert_array_equal( + np.frombuffer(raw, dtype="uint16").reshape(5, 4), data[2:7, 1:5] + ) + + +async def test_reference_create_exists_raises() -> None: + be = get_backend("reference") + store = MemoryStore() + meta = _array_meta() + await be.create_array(store, "a", meta, overwrite=False) + with pytest.raises(NodeExistsError): + await be.create_array(store, "a", meta, overwrite=False) + + +async def test_reference_read_metadata_missing_raises() -> None: + be = get_backend("reference") + with pytest.raises(NodeNotFoundError): + await be.read_metadata(MemoryStore(), "nope") +``` + +- [ ] **Step 2: Run it to verify failure** + +Run: `uv run pytest tests/crud/test_reference_backend.py -v` +Expected: FAIL — `KeyError: no CRUD backend registered as 'reference'` + +- [ ] **Step 3: Create `src/zarr/crud/_reference.py`** + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np + +from zarr.core.array import AsyncArray, create_codec_pipeline +from zarr.core.array_spec import ArrayConfig, ArraySpec +from zarr.core.buffer.core import NDBuffer, default_buffer_prototype +from zarr.core.common import ZARR_JSON, ZARRAY_JSON, ZATTRS_JSON +from zarr.core.group import GroupMetadata +from zarr.core.metadata.io import save_metadata +from zarr.core.metadata.v2 import ArrayV2Metadata +from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata +from zarr.crud._backend import NodeExistsError +from zarr.errors import NodeNotFoundError +from zarr.storage._common import StorePath + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + from zarr.abc.store import Store + from zarr.core.common import JSON + + +def _parse_array_metadata( + metadata: Mapping[str, JSON], +) -> ArrayV3Metadata | ArrayV2Metadata: + """Parse a metadata document into a v2 or v3 array metadata object.""" + data = dict(metadata) + if data.get("zarr_format") == 3: + return ArrayV3Metadata.from_dict(data) + return ArrayV2Metadata.from_dict(data) + + +def _native_dtype(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> np.dtype[Any]: + """Numpy dtype in native byte order (zarrs and the facade assume native).""" + return meta_obj.dtype.to_native_dtype().newbyteorder("=") + + +def _chunk_shape(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> tuple[int, ...]: + if isinstance(meta_obj, ArrayV3Metadata): + grid = meta_obj.chunk_grid + if not isinstance(grid, RegularChunkGridMetadata): + raise NotImplementedError("only regular chunk grids are supported") + return tuple(grid.chunk_shape) + return tuple(meta_obj.chunks) + + +def _array_spec( + meta_obj: ArrayV3Metadata | ArrayV2Metadata, shape: tuple[int, ...] +) -> ArraySpec: + return ArraySpec( + shape=shape, + dtype=meta_obj.dtype, + fill_value=meta_obj.fill_value, + config=ArrayConfig.from_dict({}), + prototype=default_buffer_prototype(), + ) + + +def _meta_key(path: str, zarr_format: int) -> str: + fname = ZARR_JSON if zarr_format == 3 else ZARRAY_JSON + p = path.strip("/") + return f"{p}/{fname}" if p else fname + + +class ReferenceBackend: + """Pure-Python CRUD backend wrapping zarr-python's own machinery. + + Constructs no high-level `Array` for chunk operations (it drives the codec + pipeline directly); it does reuse `AsyncArray.getitem` for multi-chunk + subset reads, which is exactly the `BasicIndexer` + codec-pipeline read path. + """ + + async def create_array( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + meta_obj = _parse_array_metadata(metadata) + await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) + + async def create_group( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + meta_obj = GroupMetadata.from_dict(dict(metadata)) + await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) + + async def _create( + self, store: Store, path: str, meta_obj: Any, zarr_format: int, *, overwrite: bool + ) -> None: + sp = StorePath(store, path.strip("/")) + proto = default_buffer_prototype() + if overwrite: + await store.delete_dir(path.strip("/")) + else: + key = _meta_key(path, zarr_format) + if await store.get(key, prototype=proto) is not None: + raise NodeExistsError(f"a node already exists at path {path!r}") + await save_metadata(sp, meta_obj, ensure_parents=True) + + async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: + from zarr.core._json import buffer_to_json_object + + proto = default_buffer_prototype() + p = path.strip("/") + sp = StorePath(store, p) + buf = await (sp / ZARR_JSON).get(prototype=proto) + if buf is not None: + return buffer_to_json_object(buf) + buf2 = await (sp / ZARRAY_JSON).get(prototype=proto) + if buf2 is not None: + doc = buffer_to_json_object(buf2) + zattrs = await (sp / ZATTRS_JSON).get(prototype=proto) + if zattrs is not None: + doc["attributes"] = buffer_to_json_object(zattrs) + return doc + raise NodeNotFoundError(f"no node found at path {path!r}") + + async def read_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> bytes: + meta_obj = _parse_array_metadata(metadata) + shape = _chunk_shape(meta_obj) + np_dtype = _native_dtype(meta_obj) + sp = StorePath(store, path.strip("/")) + chunk_key = meta_obj.encode_chunk_key(coords) + buf = await (sp / chunk_key).get(prototype=default_buffer_prototype()) + if buf is None: + arr = np.full(shape, meta_obj.fill_value, dtype=np_dtype) + else: + pipeline = create_codec_pipeline(meta_obj) + spec = _array_spec(meta_obj, shape) + decoded = list(await pipeline.decode_batch([(buf, spec)])) + arr = np.asarray(decoded[0].as_numpy_array(), dtype=np_dtype) + return np.ascontiguousarray(arr).tobytes() + + async def read_subset( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + start: Sequence[int], + shape: Sequence[int], + ) -> bytes: + meta_obj = _parse_array_metadata(metadata) + np_dtype = _native_dtype(meta_obj) + async_arr = AsyncArray(metadata=meta_obj, store_path=StorePath(store, path.strip("/"))) + selection = tuple(slice(s, s + length) for s, length in zip(start, shape, strict=True)) + result = await async_arr.getitem(selection) + return np.ascontiguousarray(np.asarray(result, dtype=np_dtype)).tobytes() + + async def write_chunk( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + coords: tuple[int, ...], + data: bytes, + ) -> None: + meta_obj = _parse_array_metadata(metadata) + shape = _chunk_shape(meta_obj) + np_dtype = _native_dtype(meta_obj) + sp = StorePath(store, path.strip("/")) + chunk_key = meta_obj.encode_chunk_key(coords) + arr = np.frombuffer(data, dtype=np_dtype).reshape(shape) + pipeline = create_codec_pipeline(meta_obj) + spec = _array_spec(meta_obj, shape) + encoded = list(await pipeline.encode_batch([(NDBuffer.from_ndarray_like(arr), spec)])) + buf = encoded[0] + if buf is None: + await (sp / chunk_key).delete() + else: + await (sp / chunk_key).set(buf) + + async def delete_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> None: + meta_obj = _parse_array_metadata(metadata) + sp = StorePath(store, path.strip("/")) + await (sp / meta_obj.encode_chunk_key(coords)).delete() + + async def delete_node(self, store: Store, path: str) -> None: + proto = default_buffer_prototype() + p = path.strip("/") + sp = StorePath(store, p) + present = ( + await (sp / ZARR_JSON).get(prototype=proto) is not None + or await (sp / ZARRAY_JSON).get(prototype=proto) is not None + ) + if not present: + raise NodeNotFoundError(f"no node found at path {path!r}") + await store.delete_dir(p) + + async def list_children( + self, store: Store, path: str + ) -> list[tuple[str, dict[str, JSON]]]: + proto = default_buffer_prototype() + p = path.strip("/") + sp = StorePath(store, p) + if ( + await (sp / ZARR_JSON).get(prototype=proto) is None + and await (sp / ZARRAY_JSON).get(prototype=proto) is None + ): + raise NodeNotFoundError(f"no node found at path {path!r}") + prefix = f"{p}/" if p else "" + children: list[tuple[str, dict[str, JSON]]] = [] + async for name in store.list_dir(prefix): + child_path = f"{p}/{name}" if p else name + child_sp = StorePath(store, child_path) + if ( + await (child_sp / ZARR_JSON).get(prototype=proto) is not None + or await (child_sp / ZARRAY_JSON).get(prototype=proto) is not None + ): + children.append((name, await self.read_metadata(store, child_path))) + return children +``` + +Notes for the implementer: +- `decode_batch`/`encode_batch` are async and return iterables — wrap in `list(...)`. +- `ArraySpec.dtype` is the `ZDType` object (`meta_obj.dtype`), **not** a numpy dtype. +- `_native_dtype` byte-swaps to native order so both backends return identical + bytes through the facade (the facade reads them with a native dtype). +- `AsyncArray(metadata=meta_obj, store_path=...)` constructs from an explicit + document without reading the store. + +- [ ] **Step 4: Register the reference backend** — append to `src/zarr/crud/__init__.py` (after the imports, before `__all__`) + +```python +from zarr.crud._reference import ReferenceBackend + +register_backend("reference", ReferenceBackend()) +``` + +and add `"ReferenceBackend"` to `__all__`. + +- [ ] **Step 5: Run the tests** + +Run: `uv run pytest tests/crud/test_reference_backend.py tests/crud/test_registry.py -v` +Expected: all PASS (the two previously-failing registry tests now pass too). + +- [ ] **Step 6: Commit** + +```bash +git add src/zarr/crud/_reference.py src/zarr/crud/__init__.py tests/crud/test_reference_backend.py +git commit -m "feat: pure-Python ReferenceBackend for zarr.crud" +``` + +--- + +### Task 3: shared facade `zarr.crud._api` + differential suite (reference backend) + +**Files:** +- Create: `src/zarr/crud/_api.py` +- Modify: `src/zarr/crud/__init__.py` +- Create: `tests/crud/conftest.py` +- Test: `tests/crud/test_crud.py` + +- [ ] **Step 1: Create `tests/crud/conftest.py`** + +```python +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +import zarr +from zarr.storage import LocalStore, MemoryStore + +if TYPE_CHECKING: + from collections.abc import AsyncIterator + from pathlib import Path + + from zarr.abc.store import Store + + +def _zarrs_available() -> bool: + try: + import _zarrs_bindings # noqa: F401 + except ImportError: + return False + return True + + +@pytest.fixture( + params=[ + "reference", + pytest.param( + "zarrs", + marks=pytest.mark.skipif( + not _zarrs_available(), reason="zarrs-bindings is not installed" + ), + ), + ] +) +def backend(request: pytest.FixtureRequest) -> str: + """A CRUD backend name. The zarrs param is skipped when the extension is absent.""" + import zarr.crud # noqa: F401 (ensures reference is registered) + + if request.param == "zarrs": + import zarr.zarrs # noqa: F401 (registers the zarrs backend) + return request.param + + +@pytest.fixture(params=["memory", "local"]) +async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncIterator[Store]: + if request.param == "memory": + s: Store = await MemoryStore.open() + else: + s = await LocalStore.open(root=tmp_path / "store") + try: + yield s + finally: + s.close() + + +def array_metadata(**kwargs: Any) -> dict[str, Any]: + """An array metadata document built via zarr-python itself.""" + params: dict[str, Any] = { + "shape": (8, 8), + "chunks": (4, 4), + "dtype": "uint16", + "zarr_format": 3, + } | kwargs + arr = zarr.create_array(store=MemoryStore(), **params) + doc = dict(arr.metadata.to_dict()) + if params["zarr_format"] == 2: + doc.pop("attributes", None) + return doc + + +def filled(store: Store, **kwargs: Any) -> tuple[np.ndarray[Any, np.dtype[Any]], dict[str, Any]]: + """Create an 8x8 array 'a', fill it with a ramp, return (data, metadata).""" + params: dict[str, Any] = {"shape": (8, 8), "chunks": (4, 4), "dtype": "uint16"} | kwargs + arr = zarr.create_array(store=store, name="a", **params) + data = np.arange(64, dtype=params["dtype"]).reshape(8, 8) + arr[:, :] = data + doc = dict(arr.metadata.to_dict()) + if params.get("zarr_format") == 2: + doc.pop("attributes", None) + return data, doc +``` + +- [ ] **Step 2: Write the failing test** — `tests/crud/test_crud.py` + +```python +from __future__ import annotations + +import copy +import json +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +import zarr +from tests.crud.conftest import array_metadata, filled +from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec +from zarr.core.buffer.core import default_buffer_prototype +from zarr.crud import ( + NodeExistsError, + create_new_array, + create_new_group, + create_overwrite_array, + create_overwrite_group, + delete_chunk, + delete_node, + list_children, + read_chunk, + read_encoded_chunk, + read_metadata, + read_region, + write_chunk, +) +from zarr.errors import NodeNotFoundError + +if TYPE_CHECKING: + from zarr.abc.store import Store + +GROUP_META: dict[str, Any] = {"zarr_format": 3, "node_type": "group", "attributes": {"answer": 42}} + + +# --- node lifecycle --- + +async def test_create_new_group(backend: str, store: Store) -> None: + await create_new_group(GROUP_META, store, "foo", backend=backend) + assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} + + +async def test_create_new_group_existing_raises(backend: str, store: Store) -> None: + await create_new_group(GROUP_META, store, "foo", backend=backend) + with pytest.raises(NodeExistsError): + await create_new_group(GROUP_META, store, "foo", backend=backend) + + +async def test_create_overwrite_group_replaces_array(backend: str, store: Store) -> None: + arr = zarr.create_array(store=store, name="foo", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + await create_overwrite_group(GROUP_META, store, "foo", backend=backend) + assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} + assert not await store.exists("foo/c/0") + + +async def test_create_new_array(backend: str, store: Store) -> None: + await create_new_array(array_metadata(), store, "arr", backend=backend) + a = zarr.open_array(store=store, path="arr", mode="r") + assert a.shape == (8, 8) + assert a.dtype == np.dtype("uint16") + + +async def test_create_new_array_v2(backend: str, store: Store) -> None: + await create_new_array(array_metadata(zarr_format=2), store, "arr", backend=backend) + assert zarr.open_array(store=store, path="arr", mode="r").metadata.zarr_format == 2 + + +async def test_create_overwrite_array(backend: str, store: Store) -> None: + zarr.create_group(store=store, path="arr") + await create_overwrite_array(array_metadata(), store, "arr", backend=backend) + assert zarr.open_array(store=store, path="arr", mode="r").shape == (8, 8) + + +async def test_read_metadata(backend: str, store: Store) -> None: + await create_new_array(array_metadata(), store, "arr", backend=backend) + observed = await read_metadata(store, "arr", backend=backend) + raw = await store.get("arr/zarr.json", prototype=default_buffer_prototype()) + assert raw is not None + assert observed == json.loads(raw.to_bytes()) + + +async def test_read_metadata_missing(backend: str, store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await read_metadata(store, "nope", backend=backend) + + +async def test_delete_node(backend: str, store: Store) -> None: + arr = zarr.create_array(store=store, name="doomed", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + await delete_node(store, "doomed", backend=backend) + assert not await store.exists("doomed/zarr.json") + assert not await store.exists("doomed/c/0") + + +async def test_delete_node_missing(backend: str, store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await delete_node(store, "nope", backend=backend) + + +async def test_list_children(backend: str, store: Store) -> None: + root = zarr.create_group(store=store) + root.create_group("sub_group", attributes={"kind": "group"}) + root.create_array("sub_array", shape=(4,), chunks=(2,), dtype="uint8") + by_path = dict(await list_children(store, "", backend=backend)) + assert set(by_path) == {"sub_group", "sub_array"} + assert by_path["sub_group"]["node_type"] == "group" + assert by_path["sub_array"]["node_type"] == "array" + assert not any(p.startswith("/") for p in by_path) + + +# --- chunk I/O --- + +@pytest.mark.parametrize("dtype", ["uint8", "int32", "float64", "u2"]) +async def test_read_chunk_differential(backend: str, store: Store, dtype: str) -> None: + data, meta = filled(store, dtype=dtype) + observed = await read_chunk(meta, store, "a", (1, 0), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 0:4]) + + +@pytest.mark.parametrize( + "compressors", [None, (GzipCodec(),), (ZstdCodec(),), (BloscCodec(cname="lz4"),)] +) +async def test_read_chunk_codecs(backend: str, store: Store, compressors: Any) -> None: + data, meta = filled(store, compressors=compressors) + observed = await read_chunk(meta, store, "a", (0, 1), backend=backend) + np.testing.assert_array_equal(observed, data[0:4, 4:8]) + + +async def test_read_chunk_v2(backend: str, store: Store) -> None: + data, meta = filled(store, dtype=" None: + data, meta = filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await read_chunk(meta, store, "a", (1, 1), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_read_chunk_missing_is_fill(backend: str, store: Store) -> None: + arr = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 + ) + meta = dict(arr.metadata.to_dict()) + observed = await read_chunk(meta, store, "a", (0, 0), backend=backend) + np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) + + +async def test_read_chunk_metadata_view(backend: str, store: Store) -> None: + data, meta = filled(store, dtype="uint16", compressors=None) + view = copy.deepcopy(meta) + view["data_type"] = "uint8" + view["shape"] = [8, 16] + view["chunk_grid"]["configuration"]["chunk_shape"] = [4, 8] + observed = await read_chunk(view, store, "a", (1, 0), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 0:4].view("uint8")) + + +async def test_read_chunk_readonly(backend: str, store: Store) -> None: + _, meta = filled(store) + observed = await read_chunk(meta, store, "a", (0, 0), backend=backend) + assert not observed.flags.writeable + + +async def test_write_chunk_differential(backend: str, store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a", backend=backend) + value = np.arange(16, dtype="uint16").reshape(4, 4) + await write_chunk(meta, store, "a", (0, 1), value, backend=backend) + np.testing.assert_array_equal(zarr.open_array(store=store, path="a", mode="r")[0:4, 4:8], value) + + +async def test_write_chunk_shape_mismatch(backend: str, store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a", backend=backend) + with pytest.raises(ValueError, match="chunk shape"): + await write_chunk(meta, store, "a", (0, 0), np.zeros((2, 2), dtype="uint16"), backend=backend) + + +async def test_delete_chunk(backend: str, store: Store) -> None: + data, meta = filled(store) + assert await store.exists("a/c/0/0") + await delete_chunk(meta, store, "a", (0, 0), backend=backend) + assert not await store.exists("a/c/0/0") + + +async def test_read_encoded_chunk_matches_store(backend: str, store: Store) -> None: + _, meta = filled(store) + raw = await read_encoded_chunk(meta, store, "a", (0, 0), backend=backend) + expected = await store.get("a/c/0/0", prototype=default_buffer_prototype()) + assert expected is not None + assert raw == expected.to_bytes() + + +async def test_read_encoded_chunk_missing_is_none(backend: str, store: Store) -> None: + arr = zarr.create_array(store=store, name="e", shape=(8, 8), chunks=(4, 4), dtype="uint16") + meta = dict(arr.metadata.to_dict()) + assert await read_encoded_chunk(meta, store, "e", (0, 0), backend=backend) is None + + +# --- region I/O --- + +SELECTIONS: list[Any] = [ + (slice(None), slice(None)), + (slice(2, 7), slice(1, 5)), + (slice(None), 3), + (5, slice(None)), + (3, 4), + (slice(1, 8, 2), slice(None)), + (slice(None), slice(6, 1, -2)), + (slice(-3, None), slice(None, -1)), + ..., + (..., slice(2, 4)), + (slice(0, 0), slice(None)), + (slice(2, 6),), +] + + +@pytest.mark.parametrize("sel", SELECTIONS) +async def test_read_region_differential(backend: str, store: Store, sel: Any) -> None: + data, meta = filled(store) + observed = await read_region(meta, store, "a", sel, backend=backend) + np.testing.assert_array_equal(observed, data[sel]) + + +async def test_read_region_sharding(backend: str, store: Store) -> None: + data, meta = filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await read_region(meta, store, "a", (slice(1, 7), slice(3, 8)), backend=backend) + np.testing.assert_array_equal(observed, data[1:7, 3:8]) + + +async def test_read_region_too_many_indices(backend: str, store: Store) -> None: + _, meta = filled(store) + with pytest.raises(IndexError, match="too many indices"): + await read_region(meta, store, "a", (0, 0, 0), backend=backend) + + +async def test_read_region_fancy_rejected(backend: str, store: Store) -> None: + _, meta = filled(store) + with pytest.raises(TypeError, match="only integers, slices"): + await read_region(meta, store, "a", ([0, 1], slice(None)), backend=backend) # type: ignore[arg-type] +``` + +- [ ] **Step 3: Run it to verify failure** + +Run: `uv run pytest tests/crud/test_crud.py -q` +Expected: collection error — `ImportError: cannot import name 'read_chunk' from 'zarr.crud'` + +- [ ] **Step 4: Create `src/zarr/crud/_api.py`** + +```python +from __future__ import annotations + +import operator +import types +from collections.abc import Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, cast + +import numpy as np + +from zarr.core.buffer.core import default_buffer_prototype +from zarr.crud._registry import get_backend + +if TYPE_CHECKING: + from collections.abc import Mapping + + import numpy.typing as npt + + from zarr.abc.store import Store + from zarr.core.common import JSON + from zarr.core.metadata.v2 import ArrayV2Metadata + from zarr.core.metadata.v3 import ArrayV3Metadata + from zarr.crud._backend import CrudBackend + + +@dataclass(frozen=True, slots=True) +class CrudOptions: + """Options for CRUD operations. + + Currently empty: fields (concurrency limits, checksum validation) arrive in + a later phase. Accepting it now keeps signatures stable. + """ + + +BasicIndex = int | slice | types.EllipsisType +BasicSelection = BasicIndex | tuple[BasicIndex, ...] + + +def _resolve_backend(backend: CrudBackend | str | None) -> CrudBackend: + if backend is None or isinstance(backend, str): + return get_backend(backend) + return backend + + +def _parse_array_metadata( + metadata: Mapping[str, JSON], +) -> ArrayV3Metadata | ArrayV2Metadata: + from zarr.core.metadata.v2 import ArrayV2Metadata + from zarr.core.metadata.v3 import ArrayV3Metadata + + data = dict(metadata) + if data.get("zarr_format") == 3: + return ArrayV3Metadata.from_dict(data) + return ArrayV2Metadata.from_dict(data) + + +def _chunk_dtype_and_shape( + metadata: Mapping[str, JSON], +) -> tuple[np.dtype[Any], tuple[int, ...]]: + """Resolve native-byte-order numpy dtype and regular chunk shape. + + Backends decode to (and encode from) the native in-memory representation, + applying any byte-order codec themselves, so the dtype is coerced to native. + """ + from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata + + meta_obj = _parse_array_metadata(metadata) + if isinstance(meta_obj, ArrayV3Metadata): + grid = meta_obj.chunk_grid + if not isinstance(grid, RegularChunkGridMetadata): + raise NotImplementedError("only regular chunk grids are supported") + chunk_shape = tuple(grid.chunk_shape) + else: + chunk_shape = tuple(meta_obj.chunks) + return meta_obj.dtype.to_native_dtype().newbyteorder("="), chunk_shape + + +def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: + shape = metadata.get("shape") + if not isinstance(shape, Sequence) or isinstance(shape, str): + raise TypeError("metadata document has no valid 'shape'") + result: list[int] = [] + for s in shape: + if not isinstance(s, (int, float)): + raise TypeError(f"shape element {s!r} is not a number") + if isinstance(s, float) and not s.is_integer(): + raise TypeError(f"shape element {s!r} is not an integer") + result.append(int(s)) + return tuple(result) + + +def _chunk_key(metadata: Mapping[str, JSON], path: str, coords: tuple[int, ...]) -> str: + meta_obj = _parse_array_metadata(metadata) + rel = meta_obj.encode_chunk_key(coords) + p = path.strip("/") + return f"{p}/{rel}" if p else rel + + +def _normalize_selection( + selection: BasicSelection, shape: tuple[int, ...] +) -> tuple[list[int], list[int], tuple[slice | int, ...]]: + """Normalize a numpy basic-indexing selection to a step-1 bounding box. + + Returns `(start, bounding_shape, post_index)`: the box to fetch and the + numpy index to apply to it (strides, reversals, integer-axis removal). Only + integers, slices, and `Ellipsis` are supported; fancy indexing raises. + """ + sel_tuple = selection if isinstance(selection, tuple) else (selection,) + + n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) + if n_ellipsis > 1: + raise IndexError("an index can only have a single ellipsis ('...')") + if n_ellipsis == 1: + i = sel_tuple.index(Ellipsis) + n_fill = len(shape) - (len(sel_tuple) - 1) + if n_fill < 0: + raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") + sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] + if len(sel_tuple) > len(shape): + raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") + sel_tuple = sel_tuple + (slice(None),) * (len(shape) - len(sel_tuple)) + + starts: list[int] = [] + lengths: list[int] = [] + post: list[slice | int] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + start, stop, step = sel.indices(size) + n = len(range(start, stop, step)) + if n == 0: + starts.append(0) + lengths.append(0) + post.append(slice(None)) + elif step > 0: + last = start + (n - 1) * step + starts.append(start) + lengths.append(last - start + 1) + post.append(slice(None, None, step)) + else: + last = start + (n - 1) * step + starts.append(last) + lengths.append(start - last + 1) + post.append(slice(None, None, step)) + else: + assert not isinstance(sel, types.EllipsisType), "Ellipsis already expanded above" + try: + idx = operator.index(sel) + except TypeError: + raise TypeError( + "unsupported selection element " + f"{sel!r}: only integers, slices, and Ellipsis are supported" + ) from None + if idx < 0: + idx += size + if not 0 <= idx < size: + raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") + starts.append(idx) + lengths.append(1) + post.append(0) + return starts, lengths, tuple(post) + + +# --- node lifecycle --- + +async def create_new_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create a group from a group metadata document. Raises `NodeExistsError` + if a node already exists at `path`. Not atomic against concurrent writers.""" + await _resolve_backend(backend).create_group(store, path, metadata, overwrite=False) + + +async def create_overwrite_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create a group, deleting any existing node (and children) first. Not + atomic against concurrent writers.""" + await _resolve_backend(backend).create_group(store, path, metadata, overwrite=True) + + +async def create_new_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create an array from a v2 or v3 metadata document. Raises + `NodeExistsError` if a node already exists. Not atomic against concurrent + writers.""" + await _resolve_backend(backend).create_array(store, path, metadata, overwrite=False) + + +async def create_overwrite_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create an array, deleting any existing node (and children) first. Not + atomic against concurrent writers.""" + await _resolve_backend(backend).create_array(store, path, metadata, overwrite=True) + + +async def read_metadata( + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> dict[str, JSON]: + """Read the metadata document of the array or group at `path`. Raises + `zarr.errors.NodeNotFoundError` if no node exists there.""" + return await _resolve_backend(backend).read_metadata(store, path) + + +async def delete_node( + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Delete the node at `path` and everything under it. Raises + `zarr.errors.NodeNotFoundError` if absent. `path=""` clears the store.""" + await _resolve_backend(backend).delete_node(store, path) + + +async def list_children( + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> list[tuple[str, dict[str, JSON]]]: + """List the direct children of the group at `path` as + `(path, metadata_document)` pairs (store-relative, no leading `/`). Raises + `zarr.errors.NodeNotFoundError` if no group exists there.""" + return await _resolve_backend(backend).list_children(store, path) + + +# --- chunk I/O --- + +async def read_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode the whole chunk at `chunk_coords`. The metadata document + is authoritative; missing chunks decode to the fill value. The result is a + read-only view (`.copy()` for a writable array).""" + be = _resolve_backend(backend) + raw = await be.read_chunk(store, path, metadata, tuple(chunk_coords)) + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + return np.frombuffer(raw, dtype=dtype).reshape(chunk_shape) + + +async def read_encoded_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> bytes | None: + """Read the raw, still-encoded bytes of the chunk at `chunk_coords`, or + `None` if absent. Pure store I/O (`store.get` on the chunk key): the + `backend` argument is accepted for signature uniformity but unused.""" + key = _chunk_key(metadata, path, tuple(chunk_coords)) + buf = await store.get(key, prototype=default_buffer_prototype()) + return None if buf is None else buf.to_bytes() + + +async def write_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + value: npt.ArrayLike, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Encode `value` with the codecs in `metadata` and store it as the chunk at + `chunk_coords`. `value` must match the chunk shape exactly.""" + be = _resolve_backend(backend) + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + arr = np.ascontiguousarray(np.asarray(value, dtype=dtype)) + if arr.shape != chunk_shape: + raise ValueError(f"value shape {arr.shape} does not match chunk shape {chunk_shape}") + await be.write_chunk(store, path, metadata, tuple(chunk_coords), arr.tobytes()) + + +async def delete_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Delete the chunk at `chunk_coords`. Deleting a missing chunk is a no-op.""" + await _resolve_backend(backend).delete_chunk(store, path, metadata, tuple(chunk_coords)) + + +# --- region I/O --- + +async def read_region( + metadata: Mapping[str, JSON], + store: Store, + path: str, + selection: BasicSelection, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode a region given by a numpy basic-indexing `selection` + (integers, slices with steps, `Ellipsis`). One backend call fetches the + step-1 bounding box; strides/reversals/integer-axis removal are applied as + numpy views. Missing chunks decode to the fill value. Fancy indexing raises + `TypeError`. The result is a read-only view. + + Note: a `slice(0, N, step)` reads `O(N)` bytes even though `O(N / step)` are + returned; for sparse selections over large arrays prefer `read_chunk`.""" + be = _resolve_backend(backend) + dtype, _ = _chunk_dtype_and_shape(metadata) + shape = _array_shape(metadata) + starts, lengths, post_index = _normalize_selection(selection, shape) + if 0 in lengths: + block = np.empty(lengths, dtype=dtype) + block.flags.writeable = False + else: + raw = await be.read_subset(store, path, metadata, tuple(starts), tuple(lengths)) + block = np.frombuffer(raw, dtype=dtype).reshape(lengths) + return cast("np.ndarray[Any, np.dtype[Any]]", block[post_index]) +``` + +Note: `BackendArg` is a documentation alias only; use the literal +`CrudBackend | str | None` annotations as written above. + +- [ ] **Step 5: Export the facade from `src/zarr/crud/__init__.py`** + +Add to the imports and `__all__` (keep `__all__` sorted): + +```python +from zarr.crud._api import ( + CrudOptions, + create_new_array, + create_new_group, + create_overwrite_array, + create_overwrite_group, + delete_chunk, + delete_node, + list_children, + read_chunk, + read_encoded_chunk, + read_metadata, + read_region, + write_chunk, +) +``` + +Final `__all__`: + +```python +__all__ = [ + "CrudBackend", + "CrudOptions", + "NodeExistsError", + "ReferenceBackend", + "create_new_array", + "create_new_group", + "create_overwrite_array", + "create_overwrite_group", + "delete_chunk", + "delete_node", + "get_backend", + "list_children", + "read_chunk", + "read_encoded_chunk", + "read_metadata", + "read_region", + "register_backend", + "write_chunk", +] +``` + +- [ ] **Step 6: Run the suite against the reference backend** + +Run: `uv run pytest tests/crud/test_crud.py -q` +Expected: all PASS. The `backend` fixture's `zarrs` param is skipped (no `--group zarrs`), so every test runs once on `reference` × {memory, local}. If `test_read_chunk_differential[>u2-...]` fails, the byte-order coercion in `_reference._native_dtype` / `_chunk_dtype_and_shape` is wrong — both must end in `.newbyteorder("=")`; do not weaken the assertion. + +- [ ] **Step 7: Commit** + +```bash +git add src/zarr/crud/_api.py src/zarr/crud/__init__.py tests/crud/conftest.py tests/crud/test_crud.py +git commit -m "feat: zarr.crud shared facade + differential suite (reference backend)" +``` + +--- + +### Task 4: `ZarrsBackend` + shrink `zarr.zarrs` + migrate zarrs tests + +**Files:** +- Create: `src/zarr/zarrs/_backend.py` +- Modify: `src/zarr/zarrs/__init__.py` +- Delete: `src/zarr/zarrs/_api.py` +- Delete: `tests/zarrs/test_node.py`, `tests/zarrs/test_chunk.py`, `tests/zarrs/test_api.py` +- Modify: `tests/zarrs/test_cache.py` + +- [ ] **Step 1: Create `src/zarr/zarrs/_backend.py`** + +```python +from __future__ import annotations + +import asyncio +import json +from contextlib import contextmanager +from typing import TYPE_CHECKING, cast + +import _zarrs_bindings as _zb + +from zarr.crud import NodeExistsError +from zarr.errors import NodeNotFoundError +from zarr.zarrs._bridge import resolve_store + +if TYPE_CHECKING: + from collections.abc import Iterator, Mapping, Sequence + + from zarr.abc.store import Store + from zarr.core.common import JSON + + +def _node_path(path: str) -> str: + """Convert a zarr path (`""`, `"foo/bar"`) to a zarrs node path (`"/"`, + `"/foo/bar"`).""" + return f"/{path.strip('/')}" + + +@contextmanager +def _translate_errors() -> Iterator[None]: + try: + yield + except _zb.NodeNotFoundError as err: + raise NodeNotFoundError(str(err)) from err + except _zb.NodeExistsError as err: + raise NodeExistsError(str(err)) from err + + +class ZarrsBackend: + """CRUD backend backed by the Rust `zarrs` crate via `_zarrs_bindings`. + + Owns the zarrs-specific plumbing: JSON-serializing the metadata document, + the `/`-prefixed node-path form, store resolution, offloading the blocking + Rust calls to a worker thread, and translating binding exceptions to the + canonical `zarr.crud` / `zarr.errors` types. + """ + + async def create_array( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + with _translate_errors(): + await asyncio.to_thread( + _zb.create_array, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + overwrite, + ) + + async def create_group( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + with _translate_errors(): + await asyncio.to_thread( + _zb.create_group, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + overwrite, + ) + + async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: + with _translate_errors(): + raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) + return cast("dict[str, JSON]", json.loads(raw)) + + async def read_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> bytes: + return await asyncio.to_thread( + _zb.retrieve_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(coords), + ) + + async def read_subset( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + start: Sequence[int], + shape: Sequence[int], + ) -> bytes: + return await asyncio.to_thread( + _zb.retrieve_array_subset, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(start), + list(shape), + ) + + async def write_chunk( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + coords: tuple[int, ...], + data: bytes, + ) -> None: + await asyncio.to_thread( + _zb.store_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(coords), + data, + ) + + async def delete_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> None: + await asyncio.to_thread( + _zb.erase_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(coords), + ) + + async def delete_node(self, store: Store, path: str) -> None: + with _translate_errors(): + await asyncio.to_thread(_zb.delete_node, resolve_store(store), _node_path(path)) + + async def list_children( + self, store: Store, path: str + ) -> list[tuple[str, dict[str, JSON]]]: + with _translate_errors(): + raw: list[tuple[str, str]] = await asyncio.to_thread( + _zb.list_children, resolve_store(store), _node_path(path) + ) + return [ + (child_path.lstrip("/"), cast("dict[str, JSON]", json.loads(doc))) + for child_path, doc in raw + ] +``` + +- [ ] **Step 2: Rewrite `src/zarr/zarrs/__init__.py`** + +```python +""" +The zarrs CRUD backend for `zarr.crud`, backed by the Rust +[`zarrs`](https://zarrs.dev) crate. + +Importing this module registers the `"zarrs"` backend. Requires the +`zarrs-bindings` extension (in-repo Rust crate; `uv sync --group zarrs`). Select +it with `zarr.config.set({"crud.backend": "zarrs"})` or per call via +`backend="zarrs"`. +""" + +try: + import _zarrs_bindings +except ImportError as e: + raise ImportError( + "zarr.zarrs requires the `zarrs-bindings` package, which is not installed. " + "It is built from the zarr-python repository: run `uv sync --group zarrs`." + ) from e + +from zarr.crud import register_backend +from zarr.zarrs._backend import ZarrsBackend + +__version__: str = _zarrs_bindings.version() + +register_backend("zarrs", ZarrsBackend()) + +__all__ = ["ZarrsBackend", "__version__"] +``` + +- [ ] **Step 3: Delete the moved module and obsolete tests** + +```bash +git rm src/zarr/zarrs/_api.py tests/zarrs/test_node.py tests/zarrs/test_chunk.py tests/zarrs/test_api.py +``` + +- [ ] **Step 4: Update `tests/zarrs/test_cache.py`** — change imports from the old `zarr.zarrs` functions to the `zarr.crud` facade with the zarrs backend. + +Replace the import block: + +```python +from zarr.zarrs import decode_chunk, encode_chunk +``` + +with: + +```python +from zarr.crud import read_chunk, write_chunk +``` + +Then in that file replace every `decode_chunk(` call with `read_chunk(` and every `encode_chunk(` call with `write_chunk(`, adding `backend="zarrs"` as the final keyword argument to each so they exercise the cached zarrs path. For example: + +```python + await read_chunk(meta, store, "a", (0, 0), backend="zarrs") +... + await write_chunk(meta, store, "a", (0, 0), new, backend="zarrs") +``` + +The cache assertions (`zb.array_cache_len()` / `zb.clear_array_cache()`) and the `import _zarrs_bindings as zb` line are unchanged. The module-level `pytest.importorskip("_zarrs_bindings", ...)` stays. + +- [ ] **Step 5: Add the zarrs param coverage — already wired** + +`tests/crud/conftest.py` already parametrizes `backend` over `["reference", "zarrs"]` with the zarrs case skipped when the extension is missing. No change needed; running with `--group zarrs` now exercises it. + +- [ ] **Step 6: Run everything with the zarrs extension** + +Run: `uv run --group zarrs pytest tests/crud tests/zarrs -q` +Expected: all PASS. `tests/crud/test_crud.py` now runs each test on both `reference` and `zarrs` × {memory, local}; `tests/zarrs/test_cache.py` and `test_bridge.py` pass. If a differential test passes on `reference` but fails on `zarrs` (or vice versa), the two backends disagree — investigate the backend, never weaken the assertion. + +- [ ] **Step 7: Run without the extension (reference-only path stays green)** + +Run: `uv run pytest tests/crud -q` +Expected: all PASS, zarrs params skipped. (`tests/zarrs` is not collectable without the extension; that's fine — its module-level `importorskip` skips it.) + +- [ ] **Step 8: Commit** + +```bash +git add src/zarr/zarrs tests/zarrs +git commit -m "feat: ZarrsBackend conforms to CrudBackend; zarr.zarrs is now a backend" +``` + +--- + +### Task 5: changelog, CI, and final verification + +**Files:** +- Modify: `changes/+zarrs-bindings.feature.md` +- Modify: `.github/workflows/zarrs.yml` + +- [ ] **Step 1: Reword the changelog fragment** — overwrite `changes/+zarrs-bindings.feature.md` + +```markdown +Added `zarr.crud`, an experimental backend-agnostic low-level functional API for +zarr hierarchy CRUD (`create_*`, `read_chunk`, `read_region`, `read_encoded_chunk`, +`write_chunk`, `delete_chunk`, `read_metadata`, `delete_node`, `list_children`). +Array routines take an explicit metadata document, enabling read-only views. +Operations delegate to a pluggable `CrudBackend`: a pure-Python reference backend +(the default) or the zarrs-accelerated backend in `zarr.zarrs`, backed by the Rust +[zarrs](https://zarrs.dev) crate via the in-repo `zarrs-bindings` PyO3 crate. +Select a backend with the `crud.backend` config key or a per-call `backend=` +argument. Build the zarrs backend for development with `uv sync --group zarrs`. +``` + +- [ ] **Step 2: Update the CI test command** — `.github/workflows/zarrs.yml` + +Change the test step's `run:` from: + +```yaml + run: uv run --group zarrs pytest tests/zarrs -v +``` + +to: + +```yaml + run: uv run --group zarrs pytest tests/crud tests/zarrs -v +``` + +Validate: `uvx zizmor .github/workflows/zarrs.yml` → no findings. + +- [ ] **Step 3: Lint and type-check the new code** + +Run: `uv run --group dev ruff format src/zarr/crud src/zarr/zarrs tests/crud tests/zarrs` +Run: `uv run --group dev ruff check --fix src/zarr/crud src/zarr/zarrs tests/crud tests/zarrs` +Run: `uv run --group dev --group zarrs mypy src/zarr/crud src/zarr/zarrs tests/crud tests/zarrs` +Expected: all clean. (mypy is strict; the facade and backends are fully annotated.) + +- [ ] **Step 4: Full suites, both with and without the extension** + +Run: `uv run --group zarrs pytest tests/crud tests/zarrs -q` → all pass +Run: `uv run pytest tests/crud -q` → all pass (zarrs skipped) +Run (regression — the rest of zarr-python is untouched): `uv run pytest tests/test_array.py tests/test_group.py -q` → pass + +- [ ] **Step 5: Commit** + +```bash +git add changes/+zarrs-bindings.feature.md .github/workflows/zarrs.yml +git commit -m "docs/ci: zarr.crud changelog and CI coverage" +``` + +--- + +## Out of scope (per spec) + +- Wiring `zarr.crud` under zarr-python's `Array`/`Group` classes. +- Entrypoint-based backend discovery (registration is explicit/import-time). +- A write-side region operation (`write_region`). +- Renaming the Rust `_zarrs_bindings` pyfunctions (private; adapted by `ZarrsBackend`). +- `CrudOptions` fields (concurrency, checksums) — still a placeholder. From c2e11e105dfb1aded33afaa6c6c8e144ecf5ff89 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 13:52:01 +0200 Subject: [PATCH 31/75] =?UTF-8?q?feat:=20zarr.crud=20skeleton=20=E2=80=94?= =?UTF-8?q?=20CrudBackend=20protocol,=20registry,=20config?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- src/zarr/core/config.py | 1 + src/zarr/crud/__init__.py | 23 +++++++++++++ src/zarr/crud/_backend.py | 68 +++++++++++++++++++++++++++++++++++++ src/zarr/crud/_registry.py | 29 ++++++++++++++++ tests/crud/__init__.py | 0 tests/crud/test_registry.py | 39 +++++++++++++++++++++ 6 files changed, 160 insertions(+) create mode 100644 src/zarr/crud/__init__.py create mode 100644 src/zarr/crud/_backend.py create mode 100644 src/zarr/crud/_registry.py create mode 100644 tests/crud/__init__.py create mode 100644 tests/crud/test_registry.py diff --git a/src/zarr/core/config.py b/src/zarr/core/config.py index 7dcbc78e31..288f56de69 100644 --- a/src/zarr/core/config.py +++ b/src/zarr/core/config.py @@ -107,6 +107,7 @@ def enable_gpu(self) -> ConfigSet: "path": "zarr.core.codec_pipeline.BatchedCodecPipeline", "batch_size": 1, }, + "crud": {"backend": "reference"}, "codecs": { "blosc": "zarr.codecs.blosc.BloscCodec", "gzip": "zarr.codecs.gzip.GzipCodec", diff --git a/src/zarr/crud/__init__.py b/src/zarr/crud/__init__.py new file mode 100644 index 0000000000..96eaf7af1f --- /dev/null +++ b/src/zarr/crud/__init__.py @@ -0,0 +1,23 @@ +""" +Backend-agnostic low-level functional CRUD API for zarr hierarchies. + +The public functions delegate byte- and metadata-level work to a `CrudBackend`. +Two backends ship: a pure-Python reference backend (the default) and a +zarrs-accelerated backend (`zarr.zarrs`, requires the `zarrs-bindings` +extension). Select one with the `crud.backend` config key or a per-call +`backend=` argument. + +Array routines take an explicit metadata document (a `dict` matching the +`zarr.json` / `.zarray` document) rather than reading it from the store, which +makes read-only and virtual views possible. +""" + +from zarr.crud._backend import CrudBackend, NodeExistsError +from zarr.crud._registry import get_backend, register_backend + +__all__ = [ + "CrudBackend", + "NodeExistsError", + "get_backend", + "register_backend", +] diff --git a/src/zarr/crud/_backend.py b/src/zarr/crud/_backend.py new file mode 100644 index 0000000000..808fc52498 --- /dev/null +++ b/src/zarr/crud/_backend.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Protocol, runtime_checkable + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + from zarr.abc.store import Store + from zarr.core.common import JSON + + +class NodeExistsError(ValueError): + """Raised when a node already exists at a path and overwrite was not requested.""" + + +@runtime_checkable +class CrudBackend(Protocol): + """The byte/metadata-level contract a CRUD backend must implement. + + Methods take neutral types: the metadata document as a `dict`, a zarr + `Store`, and plain zarr paths (`""`, `"foo/bar"`). They return raw bytes, + parsed JSON documents, or `None`. The shared `zarr.crud` facade builds the + numpy- and selection-level API on top of these. + + `create_*` raise `zarr.crud.NodeExistsError` when a node exists and + `overwrite` is false. `read_metadata`/`delete_node`/`list_children` raise + `zarr.errors.NodeNotFoundError` when the target is missing. + """ + + async def create_array( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: ... + + async def create_group( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: ... + + async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: ... + + async def read_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> bytes: ... + + async def read_subset( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + start: Sequence[int], + shape: Sequence[int], + ) -> bytes: ... + + async def write_chunk( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + coords: tuple[int, ...], + data: bytes, + ) -> None: ... + + async def delete_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> None: ... + + async def delete_node(self, store: Store, path: str) -> None: ... + + async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: ... diff --git a/src/zarr/crud/_registry.py b/src/zarr/crud/_registry.py new file mode 100644 index 0000000000..eca86df372 --- /dev/null +++ b/src/zarr/crud/_registry.py @@ -0,0 +1,29 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr.core.config import config + +if TYPE_CHECKING: + from zarr.crud._backend import CrudBackend + +_BACKENDS: dict[str, CrudBackend] = {} + + +def register_backend(name: str, backend: CrudBackend) -> None: + """Register a CRUD backend instance under `name`.""" + _BACKENDS[name] = backend + + +def get_backend(name: str | None = None) -> CrudBackend: + """Resolve a backend by name, or the configured default when `name` is None. + + Selecting `"zarrs"` imports `zarr.zarrs` if needed so it can self-register. + """ + if name is None: + name = config.get("crud.backend") + if name not in _BACKENDS and name == "zarrs": + import zarr.zarrs # noqa: F401 (import registers the zarrs backend) + if name not in _BACKENDS: + raise KeyError(f"no CRUD backend registered as {name!r}; registered: {sorted(_BACKENDS)}") + return _BACKENDS[name] diff --git a/tests/crud/__init__.py b/tests/crud/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/crud/test_registry.py b/tests/crud/test_registry.py new file mode 100644 index 0000000000..f5a8f8b829 --- /dev/null +++ b/tests/crud/test_registry.py @@ -0,0 +1,39 @@ +from __future__ import annotations + +import pytest + +from zarr.crud import CrudBackend, NodeExistsError, get_backend, register_backend + + +def test_node_exists_error_is_value_error() -> None: + assert issubclass(NodeExistsError, ValueError) + + +def test_default_backend_is_reference() -> None: + # the reference backend is registered at import and is the configured default + be = get_backend() + assert be is get_backend("reference") + + +def test_get_unknown_backend_raises() -> None: + with pytest.raises(KeyError, match="no CRUD backend"): + get_backend("does-not-exist") + + +def test_register_and_resolve_instance() -> None: + class Dummy: + pass + + dummy = Dummy() + register_backend("dummy-test", dummy) # type: ignore[arg-type] + try: + assert get_backend("dummy-test") is dummy # type: ignore[comparison-overlap] + finally: + from zarr.crud import _registry + + _registry._BACKENDS.pop("dummy-test", None) + + +def test_protocol_is_runtime_checkable() -> None: + # ReferenceBackend (registered as "reference") structurally satisfies the protocol + assert isinstance(get_backend("reference"), CrudBackend) From 29d126259cec41aa2e6505f55a11278bf6dbc0db Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 15:21:07 +0200 Subject: [PATCH 32/75] fix: friendly error for missing zarrs backend; document protocol limits - wrap the lazy `import zarr.zarrs` in `get_backend` with a try/except that raises a helpful ImportError when the zarrs-bindings extension is not installed - add a thread-safety comment above the `_BACKENDS` dict explaining that CPython's import lock + GIL make it safe without additional locking - document the `runtime_checkable` limitation in the `CrudBackend` docstring: isinstance only checks method names, not signatures or async-ness; mypy is the authoritative conformance check Co-Authored-By: Claude Fable 5 --- src/zarr/crud/_backend.py | 4 ++++ src/zarr/crud/_registry.py | 13 ++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/src/zarr/crud/_backend.py b/src/zarr/crud/_backend.py index 808fc52498..79fa1f49df 100644 --- a/src/zarr/crud/_backend.py +++ b/src/zarr/crud/_backend.py @@ -25,6 +25,10 @@ class CrudBackend(Protocol): `create_*` raise `zarr.crud.NodeExistsError` when a node exists and `overwrite` is false. `read_metadata`/`delete_node`/`list_children` raise `zarr.errors.NodeNotFoundError` when the target is missing. + + Note: because this protocol is `runtime_checkable`, `isinstance` checks only + verify that the method names exist, not their signatures or that they are + async. Static type checking (mypy) is the authoritative conformance check. """ async def create_array( diff --git a/src/zarr/crud/_registry.py b/src/zarr/crud/_registry.py index eca86df372..84fde1bc20 100644 --- a/src/zarr/crud/_registry.py +++ b/src/zarr/crud/_registry.py @@ -7,6 +7,9 @@ if TYPE_CHECKING: from zarr.crud._backend import CrudBackend +# Backends are registered at import time (reference by zarr.crud, zarrs by +# zarr.zarrs). CPython's import lock plus the GIL make this dict safe without +# additional locking. _BACKENDS: dict[str, CrudBackend] = {} @@ -23,7 +26,15 @@ def get_backend(name: str | None = None) -> CrudBackend: if name is None: name = config.get("crud.backend") if name not in _BACKENDS and name == "zarrs": - import zarr.zarrs # noqa: F401 (import registers the zarrs backend) + # "reference" is pre-registered by zarr.crud at import; "zarrs" lives in a + # separate package that may not be imported yet, so load it on demand. + try: + import zarr.zarrs # noqa: F401 (import registers the zarrs backend) + except ImportError as e: + raise ImportError( + "the 'zarrs' CRUD backend requires the zarrs-bindings extension; " + "install it with: uv sync --group zarrs" + ) from e if name not in _BACKENDS: raise KeyError(f"no CRUD backend registered as {name!r}; registered: {sorted(_BACKENDS)}") return _BACKENDS[name] From 3d3784dde20b567a3eb547b153f7278d5761df85 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 15:35:17 +0200 Subject: [PATCH 33/75] feat: pure-Python ReferenceBackend for zarr.crud Implements `ReferenceBackend` in `src/zarr/crud/_reference.py`, registers it as the default `"reference"` backend at `zarr.crud` import time. Drives chunk encode/decode via `create_codec_pipeline` + the abstract `CodecPipeline.encode`/`decode` methods (not `encode_batch`/`decode_batch`, which live only on `BatchedCodecPipeline` and are not in the abstract Protocol), and multi-chunk subset reads via `AsyncArray.getitem`. Fixes two previously-failing registry tests; adds four new backend tests covering round-trip chunk I/O, multi-chunk subset reads, duplicate-create error, and missing-metadata error. Co-Authored-By: Claude Fable 5 --- src/zarr/crud/__init__.py | 4 + src/zarr/crud/_reference.py | 215 +++++++++++++++++++++++++++ tests/crud/test_reference_backend.py | 53 +++++++ 3 files changed, 272 insertions(+) create mode 100644 src/zarr/crud/_reference.py create mode 100644 tests/crud/test_reference_backend.py diff --git a/src/zarr/crud/__init__.py b/src/zarr/crud/__init__.py index 96eaf7af1f..ac6d1a5e60 100644 --- a/src/zarr/crud/__init__.py +++ b/src/zarr/crud/__init__.py @@ -13,11 +13,15 @@ """ from zarr.crud._backend import CrudBackend, NodeExistsError +from zarr.crud._reference import ReferenceBackend from zarr.crud._registry import get_backend, register_backend +register_backend("reference", ReferenceBackend()) + __all__ = [ "CrudBackend", "NodeExistsError", + "ReferenceBackend", "get_backend", "register_backend", ] diff --git a/src/zarr/crud/_reference.py b/src/zarr/crud/_reference.py new file mode 100644 index 0000000000..f770cc88b4 --- /dev/null +++ b/src/zarr/crud/_reference.py @@ -0,0 +1,215 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np + +from zarr.core.array import AsyncArray, create_codec_pipeline +from zarr.core.array_spec import ArrayConfig, ArraySpec +from zarr.core.buffer.core import NDBuffer, default_buffer_prototype +from zarr.core.common import ZARR_JSON, ZARRAY_JSON, ZATTRS_JSON +from zarr.core.group import GroupMetadata +from zarr.core.metadata.io import save_metadata +from zarr.core.metadata.v2 import ArrayV2Metadata +from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata +from zarr.crud._backend import NodeExistsError +from zarr.errors import NodeNotFoundError +from zarr.storage._common import StorePath + +if TYPE_CHECKING: + from collections.abc import Mapping, Sequence + + from zarr.abc.store import Store + from zarr.core.common import JSON + + +def _parse_array_metadata( + metadata: Mapping[str, JSON], +) -> ArrayV3Metadata | ArrayV2Metadata: + """Parse a metadata document into a v2 or v3 array metadata object.""" + data = dict(metadata) + if data.get("zarr_format") == 3: + return ArrayV3Metadata.from_dict(data) + return ArrayV2Metadata.from_dict(data) + + +def _native_dtype(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> np.dtype[Any]: + """Numpy dtype in native byte order (zarrs and the facade assume native).""" + return meta_obj.dtype.to_native_dtype().newbyteorder("=") + + +def _chunk_shape(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> tuple[int, ...]: + if isinstance(meta_obj, ArrayV3Metadata): + grid = meta_obj.chunk_grid + if not isinstance(grid, RegularChunkGridMetadata): + raise NotImplementedError("only regular chunk grids are supported") + return tuple(grid.chunk_shape) + return tuple(meta_obj.chunks) + + +def _array_spec(meta_obj: ArrayV3Metadata | ArrayV2Metadata, shape: tuple[int, ...]) -> ArraySpec: + return ArraySpec( + shape=shape, + dtype=meta_obj.dtype, + fill_value=meta_obj.fill_value, + config=ArrayConfig.from_dict({}), + prototype=default_buffer_prototype(), + ) + + +def _meta_key(path: str, zarr_format: int) -> str: + fname = ZARR_JSON if zarr_format == 3 else ZARRAY_JSON + p = path.strip("/") + return f"{p}/{fname}" if p else fname + + +class ReferenceBackend: + """Pure-Python CRUD backend wrapping zarr-python's own machinery. + + Constructs no high-level `Array` for chunk operations (it drives the codec + pipeline directly); it does reuse `AsyncArray.getitem` for multi-chunk + subset reads, which is exactly the `BasicIndexer` + codec-pipeline read path. + """ + + async def create_array( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + meta_obj = _parse_array_metadata(metadata) + await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) + + async def create_group( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + meta_obj = GroupMetadata.from_dict(dict(metadata)) + await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) + + async def _create( + self, store: Store, path: str, meta_obj: Any, zarr_format: int, *, overwrite: bool + ) -> None: + sp = StorePath(store, path.strip("/")) + proto = default_buffer_prototype() + if overwrite: + await store.delete_dir(path.strip("/")) + else: + key = _meta_key(path, zarr_format) + if await store.get(key, prototype=proto) is not None: + raise NodeExistsError(f"a node already exists at path {path!r}") + await save_metadata(sp, meta_obj, ensure_parents=True) + + async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: + from zarr.core._json import buffer_to_json_object + + proto = default_buffer_prototype() + p = path.strip("/") + sp = StorePath(store, p) + buf = await (sp / ZARR_JSON).get(prototype=proto) + if buf is not None: + return buffer_to_json_object(buf) + buf2 = await (sp / ZARRAY_JSON).get(prototype=proto) + if buf2 is not None: + doc = buffer_to_json_object(buf2) + zattrs = await (sp / ZATTRS_JSON).get(prototype=proto) + if zattrs is not None: + doc["attributes"] = buffer_to_json_object(zattrs) + return doc + raise NodeNotFoundError(f"no node found at path {path!r}") + + async def read_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> bytes: + meta_obj = _parse_array_metadata(metadata) + shape = _chunk_shape(meta_obj) + np_dtype = _native_dtype(meta_obj) + sp = StorePath(store, path.strip("/")) + chunk_key = meta_obj.encode_chunk_key(coords) + buf = await (sp / chunk_key).get(prototype=default_buffer_prototype()) + if buf is None: + arr = np.full(shape, meta_obj.fill_value, dtype=np_dtype) + else: + pipeline = create_codec_pipeline(meta_obj) + spec = _array_spec(meta_obj, shape) + decoded = list(await pipeline.decode([(buf, spec)])) + nd_buf = decoded[0] + if nd_buf is None: + arr = np.full(shape, meta_obj.fill_value, dtype=np_dtype) + else: + arr = np.asarray(nd_buf.as_numpy_array(), dtype=np_dtype) + return np.ascontiguousarray(arr).tobytes() + + async def read_subset( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + start: Sequence[int], + shape: Sequence[int], + ) -> bytes: + meta_obj = _parse_array_metadata(metadata) + np_dtype = _native_dtype(meta_obj) + async_arr = AsyncArray(metadata=meta_obj, store_path=StorePath(store, path.strip("/"))) + selection = tuple(slice(s, s + length) for s, length in zip(start, shape, strict=True)) + result = await async_arr.getitem(selection) + return np.ascontiguousarray(np.asarray(result, dtype=np_dtype)).tobytes() + + async def write_chunk( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + coords: tuple[int, ...], + data: bytes, + ) -> None: + meta_obj = _parse_array_metadata(metadata) + shape = _chunk_shape(meta_obj) + np_dtype = _native_dtype(meta_obj) + sp = StorePath(store, path.strip("/")) + chunk_key = meta_obj.encode_chunk_key(coords) + arr = np.frombuffer(data, dtype=np_dtype).reshape(shape) + pipeline = create_codec_pipeline(meta_obj) + spec = _array_spec(meta_obj, shape) + encoded = list(await pipeline.encode([(NDBuffer.from_ndarray_like(arr), spec)])) + buf = encoded[0] + if buf is None: + await (sp / chunk_key).delete() + else: + await (sp / chunk_key).set(buf) + + async def delete_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> None: + meta_obj = _parse_array_metadata(metadata) + sp = StorePath(store, path.strip("/")) + await (sp / meta_obj.encode_chunk_key(coords)).delete() + + async def delete_node(self, store: Store, path: str) -> None: + proto = default_buffer_prototype() + p = path.strip("/") + sp = StorePath(store, p) + present = ( + await (sp / ZARR_JSON).get(prototype=proto) is not None + or await (sp / ZARRAY_JSON).get(prototype=proto) is not None + ) + if not present: + raise NodeNotFoundError(f"no node found at path {path!r}") + await store.delete_dir(p) + + async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: + proto = default_buffer_prototype() + p = path.strip("/") + sp = StorePath(store, p) + if ( + await (sp / ZARR_JSON).get(prototype=proto) is None + and await (sp / ZARRAY_JSON).get(prototype=proto) is None + ): + raise NodeNotFoundError(f"no node found at path {path!r}") + prefix = f"{p}/" if p else "" + children: list[tuple[str, dict[str, JSON]]] = [] + async for name in store.list_dir(prefix): + child_path = f"{p}/{name}" if p else name + child_sp = StorePath(store, child_path) + if ( + await (child_sp / ZARR_JSON).get(prototype=proto) is not None + or await (child_sp / ZARRAY_JSON).get(prototype=proto) is not None + ): + children.append((name, await self.read_metadata(store, child_path))) + return children diff --git a/tests/crud/test_reference_backend.py b/tests/crud/test_reference_backend.py new file mode 100644 index 0000000000..9b6f4662eb --- /dev/null +++ b/tests/crud/test_reference_backend.py @@ -0,0 +1,53 @@ +from __future__ import annotations + +from typing import Any + +import numpy as np +import pytest + +import zarr +from zarr.crud import NodeExistsError, get_backend +from zarr.errors import NodeNotFoundError +from zarr.storage import MemoryStore + + +def _array_meta() -> dict[str, Any]: + arr = zarr.create_array(store=MemoryStore(), shape=(8, 8), chunks=(4, 4), dtype="uint16") + return dict(arr.metadata.to_dict()) + + +async def test_reference_round_trip_chunk() -> None: + be = get_backend("reference") + store = MemoryStore() + meta = _array_meta() + await be.create_array(store, "a", meta, overwrite=False) + value = np.arange(16, dtype="uint16").reshape(4, 4) + await be.write_chunk(store, "a", meta, (0, 1), value.tobytes()) + raw = await be.read_chunk(store, "a", meta, (0, 1)) + np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(4, 4), value) + + +async def test_reference_read_subset_spans_chunks() -> None: + be = get_backend("reference") + store = MemoryStore() + arr = zarr.create_array(store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16") + data = np.arange(64, dtype="uint16").reshape(8, 8) + arr[:, :] = data + meta = dict(arr.metadata.to_dict()) + raw = await be.read_subset(store, "a", meta, (2, 1), (5, 4)) + np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(5, 4), data[2:7, 1:5]) + + +async def test_reference_create_exists_raises() -> None: + be = get_backend("reference") + store = MemoryStore() + meta = _array_meta() + await be.create_array(store, "a", meta, overwrite=False) + with pytest.raises(NodeExistsError): + await be.create_array(store, "a", meta, overwrite=False) + + +async def test_reference_read_metadata_missing_raises() -> None: + be = get_backend("reference") + with pytest.raises(NodeNotFoundError): + await be.read_metadata(MemoryStore(), "nope") From 9197a021a45f81d3d3ff9bff666a8ee22788a3df Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 16:08:42 +0200 Subject: [PATCH 34/75] fix: honor v2 memory order in ReferenceBackend chunk codec spec `_array_spec` was hardcoding `ArrayConfig.from_dict({})`, which always defaults to C order, causing silent data corruption for v2 arrays created with `order="F"`. `read_chunk` returned transposed bytes and `write_chunk` stored bytes that zarr-python read back incorrectly. Fix: detect `ArrayV2Metadata` and pass its `.order` attribute into the `ArrayConfig`, so the codec pipeline uses the correct memory layout. v3 arrays are unaffected (order there is a transpose codec, not memory order). Co-Authored-By: Claude Fable 5 --- src/zarr/crud/_reference.py | 3 ++- tests/crud/test_reference_backend.py | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/src/zarr/crud/_reference.py b/src/zarr/crud/_reference.py index f770cc88b4..e8b76571bd 100644 --- a/src/zarr/crud/_reference.py +++ b/src/zarr/crud/_reference.py @@ -48,11 +48,12 @@ def _chunk_shape(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> tuple[int, ...] def _array_spec(meta_obj: ArrayV3Metadata | ArrayV2Metadata, shape: tuple[int, ...]) -> ArraySpec: + order = meta_obj.order if isinstance(meta_obj, ArrayV2Metadata) else "C" return ArraySpec( shape=shape, dtype=meta_obj.dtype, fill_value=meta_obj.fill_value, - config=ArrayConfig.from_dict({}), + config=ArrayConfig.from_dict({"order": order}), prototype=default_buffer_prototype(), ) diff --git a/tests/crud/test_reference_backend.py b/tests/crud/test_reference_backend.py index 9b6f4662eb..4fef43427b 100644 --- a/tests/crud/test_reference_backend.py +++ b/tests/crud/test_reference_backend.py @@ -51,3 +51,23 @@ async def test_reference_read_metadata_missing_raises() -> None: be = get_backend("reference") with pytest.raises(NodeNotFoundError): await be.read_metadata(MemoryStore(), "nope") + + +async def test_reference_v2_fortran_order_round_trip() -> None: + be = get_backend("reference") + store = MemoryStore() + arr = zarr.create_array( + store=store, name="f", shape=(4, 6), chunks=(4, 6), dtype="uint16", order="F", zarr_format=2 + ) + data = np.arange(24, dtype="uint16").reshape(4, 6) + arr[:, :] = data + meta = dict(arr.metadata.to_dict()) + meta.pop("attributes", None) + # read_chunk must return native C-contiguous bytes matching the logical data + raw = await be.read_chunk(store, "f", meta, (0, 0)) + np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(4, 6), data) + # write_chunk must store data zarr-python reads back correctly + new = (data + 100).astype("uint16") + await be.write_chunk(store, "f", meta, (0, 0), np.ascontiguousarray(new).tobytes()) + back = zarr.open_array(store=store, path="f", mode="r") + np.testing.assert_array_equal(back[:, :], new) From cc7fa45fac8958709a2024a6bed64c1f999b8bfe Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 16:20:32 +0200 Subject: [PATCH 35/75] feat: zarr.crud shared facade + differential suite (reference backend) Adds the public `zarr.crud` facade dispatching to a CrudBackend, plus the differential test suite exercising it against the reference backend. The zarrs param in the backend fixture is skipped until Task 4 registers the zarrs backend. Co-Authored-By: Claude Fable 5 --- src/zarr/crud/__init__.py | 28 +++ src/zarr/crud/_api.py | 357 ++++++++++++++++++++++++++++++++++++++ tests/crud/conftest.py | 95 ++++++++++ tests/crud/test_crud.py | 254 +++++++++++++++++++++++++++ 4 files changed, 734 insertions(+) create mode 100644 src/zarr/crud/_api.py create mode 100644 tests/crud/conftest.py create mode 100644 tests/crud/test_crud.py diff --git a/src/zarr/crud/__init__.py b/src/zarr/crud/__init__.py index ac6d1a5e60..33b01668b2 100644 --- a/src/zarr/crud/__init__.py +++ b/src/zarr/crud/__init__.py @@ -12,6 +12,21 @@ makes read-only and virtual views possible. """ +from zarr.crud._api import ( + CrudOptions, + create_new_array, + create_new_group, + create_overwrite_array, + create_overwrite_group, + delete_chunk, + delete_node, + list_children, + read_chunk, + read_encoded_chunk, + read_metadata, + read_region, + write_chunk, +) from zarr.crud._backend import CrudBackend, NodeExistsError from zarr.crud._reference import ReferenceBackend from zarr.crud._registry import get_backend, register_backend @@ -20,8 +35,21 @@ __all__ = [ "CrudBackend", + "CrudOptions", "NodeExistsError", "ReferenceBackend", + "create_new_array", + "create_new_group", + "create_overwrite_array", + "create_overwrite_group", + "delete_chunk", + "delete_node", "get_backend", + "list_children", + "read_chunk", + "read_encoded_chunk", + "read_metadata", + "read_region", "register_backend", + "write_chunk", ] diff --git a/src/zarr/crud/_api.py b/src/zarr/crud/_api.py new file mode 100644 index 0000000000..5fdb8fe1d2 --- /dev/null +++ b/src/zarr/crud/_api.py @@ -0,0 +1,357 @@ +from __future__ import annotations + +import operator +import types +from collections.abc import Sequence +from dataclasses import dataclass +from typing import TYPE_CHECKING, Any, cast + +import numpy as np + +from zarr.core.buffer.core import default_buffer_prototype +from zarr.crud._registry import get_backend + +if TYPE_CHECKING: + from collections.abc import Mapping + + import numpy.typing as npt + + from zarr.abc.store import Store + from zarr.core.common import JSON + from zarr.core.metadata.v2 import ArrayV2Metadata + from zarr.core.metadata.v3 import ArrayV3Metadata + from zarr.crud._backend import CrudBackend + + +@dataclass(frozen=True, slots=True) +class CrudOptions: + """Options for CRUD operations. + + Currently empty: fields (concurrency limits, checksum validation) arrive in + a later phase. Accepting it now keeps signatures stable. + """ + + +BasicIndex = int | slice | types.EllipsisType +BasicSelection = BasicIndex | tuple[BasicIndex, ...] + + +def _resolve_backend(backend: CrudBackend | str | None) -> CrudBackend: + if backend is None or isinstance(backend, str): + return get_backend(backend) + return backend + + +def _parse_array_metadata( + metadata: Mapping[str, JSON], +) -> ArrayV3Metadata | ArrayV2Metadata: + from zarr.core.metadata.v2 import ArrayV2Metadata + from zarr.core.metadata.v3 import ArrayV3Metadata + + data = dict(metadata) + if data.get("zarr_format") == 3: + return ArrayV3Metadata.from_dict(data) + return ArrayV2Metadata.from_dict(data) + + +def _chunk_dtype_and_shape( + metadata: Mapping[str, JSON], +) -> tuple[np.dtype[Any], tuple[int, ...]]: + """Resolve native-byte-order numpy dtype and regular chunk shape. + + Backends decode to (and encode from) the native in-memory representation, + applying any byte-order codec themselves, so the dtype is coerced to native. + """ + from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata + + meta_obj = _parse_array_metadata(metadata) + if isinstance(meta_obj, ArrayV3Metadata): + grid = meta_obj.chunk_grid + if not isinstance(grid, RegularChunkGridMetadata): + raise NotImplementedError("only regular chunk grids are supported") + chunk_shape = tuple(grid.chunk_shape) + else: + chunk_shape = tuple(meta_obj.chunks) + return meta_obj.dtype.to_native_dtype().newbyteorder("="), chunk_shape + + +def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: + shape = metadata.get("shape") + if not isinstance(shape, Sequence) or isinstance(shape, str): + raise TypeError("metadata document has no valid 'shape'") + result: list[int] = [] + for s in shape: + if not isinstance(s, (int, float)): + raise TypeError(f"shape element {s!r} is not a number") + if isinstance(s, float) and not s.is_integer(): + raise TypeError(f"shape element {s!r} is not an integer") + result.append(int(s)) + return tuple(result) + + +def _chunk_key(metadata: Mapping[str, JSON], path: str, coords: tuple[int, ...]) -> str: + meta_obj = _parse_array_metadata(metadata) + rel = meta_obj.encode_chunk_key(coords) + p = path.strip("/") + return f"{p}/{rel}" if p else rel + + +def _normalize_selection( + selection: BasicSelection, shape: tuple[int, ...] +) -> tuple[list[int], list[int], tuple[slice | int, ...]]: + """Normalize a numpy basic-indexing selection to a step-1 bounding box. + + Returns `(start, bounding_shape, post_index)`: the box to fetch and the + numpy index to apply to it (strides, reversals, integer-axis removal). Only + integers, slices, and `Ellipsis` are supported; fancy indexing raises. + """ + sel_tuple = selection if isinstance(selection, tuple) else (selection,) + + n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) + if n_ellipsis > 1: + raise IndexError("an index can only have a single ellipsis ('...')") + if n_ellipsis == 1: + i = sel_tuple.index(Ellipsis) + n_fill = len(shape) - (len(sel_tuple) - 1) + if n_fill < 0: + raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") + sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] + if len(sel_tuple) > len(shape): + raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") + sel_tuple = sel_tuple + (slice(None),) * (len(shape) - len(sel_tuple)) + + starts: list[int] = [] + lengths: list[int] = [] + post: list[slice | int] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + start, stop, step = sel.indices(size) + n = len(range(start, stop, step)) + if n == 0: + starts.append(0) + lengths.append(0) + post.append(slice(None)) + elif step > 0: + last = start + (n - 1) * step + starts.append(start) + lengths.append(last - start + 1) + post.append(slice(None, None, step)) + else: + last = start + (n - 1) * step + starts.append(last) + lengths.append(start - last + 1) + post.append(slice(None, None, step)) + else: + assert not isinstance(sel, types.EllipsisType), "Ellipsis already expanded above" + try: + idx = operator.index(sel) + except TypeError: + raise TypeError( + "unsupported selection element " + f"{sel!r}: only integers, slices, and Ellipsis are supported" + ) from None + if idx < 0: + idx += size + if not 0 <= idx < size: + raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") + starts.append(idx) + lengths.append(1) + post.append(0) + return starts, lengths, tuple(post) + + +# --- node lifecycle --- + + +async def create_new_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create a group from a group metadata document. Raises `NodeExistsError` + if a node already exists at `path`. Not atomic against concurrent writers.""" + await _resolve_backend(backend).create_group(store, path, metadata, overwrite=False) + + +async def create_overwrite_group( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create a group, deleting any existing node (and children) first. Not + atomic against concurrent writers.""" + await _resolve_backend(backend).create_group(store, path, metadata, overwrite=True) + + +async def create_new_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create an array from a v2 or v3 metadata document. Raises + `NodeExistsError` if a node already exists. Not atomic against concurrent + writers.""" + await _resolve_backend(backend).create_array(store, path, metadata, overwrite=False) + + +async def create_overwrite_array( + metadata: Mapping[str, JSON], + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Create an array, deleting any existing node (and children) first. Not + atomic against concurrent writers.""" + await _resolve_backend(backend).create_array(store, path, metadata, overwrite=True) + + +async def read_metadata( + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> dict[str, JSON]: + """Read the metadata document of the array or group at `path`. Raises + `zarr.errors.NodeNotFoundError` if no node exists there.""" + return await _resolve_backend(backend).read_metadata(store, path) + + +async def delete_node( + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Delete the node at `path` and everything under it. Raises + `zarr.errors.NodeNotFoundError` if absent. `path=""` clears the store.""" + await _resolve_backend(backend).delete_node(store, path) + + +async def list_children( + store: Store, + path: str, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> list[tuple[str, dict[str, JSON]]]: + """List the direct children of the group at `path` as + `(path, metadata_document)` pairs (store-relative, no leading `/`). Raises + `zarr.errors.NodeNotFoundError` if no group exists there.""" + return await _resolve_backend(backend).list_children(store, path) + + +# --- chunk I/O --- + + +async def read_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode the whole chunk at `chunk_coords`. The metadata document + is authoritative; missing chunks decode to the fill value. The result is a + read-only view (`.copy()` for a writable array).""" + be = _resolve_backend(backend) + raw = await be.read_chunk(store, path, metadata, tuple(chunk_coords)) + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + return np.frombuffer(raw, dtype=dtype).reshape(chunk_shape) + + +async def read_encoded_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> bytes | None: + """Read the raw, still-encoded bytes of the chunk at `chunk_coords`, or + `None` if absent. Pure store I/O (`store.get` on the chunk key): the + `backend` argument is accepted for signature uniformity but unused.""" + key = _chunk_key(metadata, path, tuple(chunk_coords)) + buf = await store.get(key, prototype=default_buffer_prototype()) + return None if buf is None else buf.to_bytes() + + +async def write_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + value: npt.ArrayLike, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Encode `value` with the codecs in `metadata` and store it as the chunk at + `chunk_coords`. `value` must match the chunk shape exactly.""" + be = _resolve_backend(backend) + dtype, chunk_shape = _chunk_dtype_and_shape(metadata) + arr = np.ascontiguousarray(np.asarray(value, dtype=dtype)) + if arr.shape != chunk_shape: + raise ValueError(f"value shape {arr.shape} does not match chunk shape {chunk_shape}") + await be.write_chunk(store, path, metadata, tuple(chunk_coords), arr.tobytes()) + + +async def delete_chunk( + metadata: Mapping[str, JSON], + store: Store, + path: str, + chunk_coords: tuple[int, ...], + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> None: + """Delete the chunk at `chunk_coords`. Deleting a missing chunk is a no-op.""" + await _resolve_backend(backend).delete_chunk(store, path, metadata, tuple(chunk_coords)) + + +# --- region I/O --- + + +async def read_region( + metadata: Mapping[str, JSON], + store: Store, + path: str, + selection: BasicSelection, + *, + options: CrudOptions | None = None, + backend: CrudBackend | str | None = None, +) -> np.ndarray[Any, np.dtype[Any]]: + """Read and decode a region given by a numpy basic-indexing `selection` + (integers, slices with steps, `Ellipsis`). One backend call fetches the + step-1 bounding box; strides/reversals/integer-axis removal are applied as + numpy views. Missing chunks decode to the fill value. Fancy indexing raises + `TypeError`. The result is a read-only view. + + Note: a `slice(0, N, step)` reads `O(N)` bytes even though `O(N / step)` are + returned; for sparse selections over large arrays prefer `read_chunk`.""" + be = _resolve_backend(backend) + dtype, _ = _chunk_dtype_and_shape(metadata) + shape = _array_shape(metadata) + starts, lengths, post_index = _normalize_selection(selection, shape) + if 0 in lengths: + block = np.empty(lengths, dtype=dtype) + block.flags.writeable = False + else: + raw = await be.read_subset(store, path, metadata, tuple(starts), tuple(lengths)) + block = np.frombuffer(raw, dtype=dtype).reshape(lengths) + return cast("np.ndarray[Any, np.dtype[Any]]", block[post_index]) diff --git a/tests/crud/conftest.py b/tests/crud/conftest.py new file mode 100644 index 0000000000..fbf2cf9e02 --- /dev/null +++ b/tests/crud/conftest.py @@ -0,0 +1,95 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +import zarr +from zarr.storage import LocalStore, MemoryStore + +if TYPE_CHECKING: + from collections.abc import AsyncIterator + from pathlib import Path + + from zarr.abc.store import Store + + +def _zarrs_available() -> bool: + """Return True only if the zarrs CrudBackend is fully usable (registered).""" + try: + import _zarrs_bindings # noqa: F401 + except ImportError: + return False + try: + import zarr.zarrs + except ImportError: + return False + # The module might exist but not yet register the zarrs CrudBackend (e.g. + # Task 4 not yet merged). Verify registration before enabling the param. + try: + import zarr.crud + + zarr.crud.get_backend("zarrs") + except (ImportError, KeyError): + return False + return True + + +@pytest.fixture( + params=[ + "reference", + pytest.param( + "zarrs", + marks=pytest.mark.skipif( + not _zarrs_available(), reason="zarrs-bindings is not installed" + ), + ), + ] +) +def backend(request: pytest.FixtureRequest) -> str: + """A CRUD backend name. The zarrs param is skipped when the extension is absent.""" + import zarr.crud + + if request.param == "zarrs": + import zarr.zarrs # noqa: F401 (registers the zarrs backend) + return str(request.param) + + +@pytest.fixture(params=["memory", "local"]) +async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncIterator[Store]: + if request.param == "memory": + s: Store = await MemoryStore.open() + else: + s = await LocalStore.open(root=tmp_path / "store") + try: + yield s + finally: + s.close() + + +def array_metadata(**kwargs: Any) -> dict[str, Any]: + """An array metadata document built via zarr-python itself.""" + params: dict[str, Any] = { + "shape": (8, 8), + "chunks": (4, 4), + "dtype": "uint16", + "zarr_format": 3, + } | kwargs + arr = zarr.create_array(store=MemoryStore(), **params) + doc = dict(arr.metadata.to_dict()) + if params["zarr_format"] == 2: + doc.pop("attributes", None) + return doc + + +def filled(store: Store, **kwargs: Any) -> tuple[np.ndarray[Any, np.dtype[Any]], dict[str, Any]]: + """Create an 8x8 array 'a', fill it with a ramp, return (data, metadata).""" + params: dict[str, Any] = {"shape": (8, 8), "chunks": (4, 4), "dtype": "uint16"} | kwargs + arr = zarr.create_array(store=store, name="a", **params) + data = np.arange(64, dtype=params["dtype"]).reshape(8, 8) + arr[:, :] = data + doc = dict(arr.metadata.to_dict()) + if params.get("zarr_format") == 2: + doc.pop("attributes", None) + return data, doc diff --git a/tests/crud/test_crud.py b/tests/crud/test_crud.py new file mode 100644 index 0000000000..c301725e22 --- /dev/null +++ b/tests/crud/test_crud.py @@ -0,0 +1,254 @@ +from __future__ import annotations + +import copy +import json +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +import zarr +from tests.crud.conftest import array_metadata, filled +from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec +from zarr.core.buffer.core import default_buffer_prototype +from zarr.crud import ( + NodeExistsError, + create_new_array, + create_new_group, + create_overwrite_array, + create_overwrite_group, + delete_chunk, + delete_node, + list_children, + read_chunk, + read_encoded_chunk, + read_metadata, + read_region, + write_chunk, +) +from zarr.errors import NodeNotFoundError + +if TYPE_CHECKING: + from zarr.abc.store import Store + +GROUP_META: dict[str, Any] = {"zarr_format": 3, "node_type": "group", "attributes": {"answer": 42}} + + +# --- node lifecycle --- + + +async def test_create_new_group(backend: str, store: Store) -> None: + await create_new_group(GROUP_META, store, "foo", backend=backend) + assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} + + +async def test_create_new_group_existing_raises(backend: str, store: Store) -> None: + await create_new_group(GROUP_META, store, "foo", backend=backend) + with pytest.raises(NodeExistsError): + await create_new_group(GROUP_META, store, "foo", backend=backend) + + +async def test_create_overwrite_group_replaces_array(backend: str, store: Store) -> None: + arr = zarr.create_array(store=store, name="foo", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + await create_overwrite_group(GROUP_META, store, "foo", backend=backend) + assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} + assert not await store.exists("foo/c/0") + + +async def test_create_new_array(backend: str, store: Store) -> None: + await create_new_array(array_metadata(), store, "arr", backend=backend) + a = zarr.open_array(store=store, path="arr", mode="r") + assert a.shape == (8, 8) + assert a.dtype == np.dtype("uint16") + + +async def test_create_new_array_v2(backend: str, store: Store) -> None: + await create_new_array(array_metadata(zarr_format=2), store, "arr", backend=backend) + assert zarr.open_array(store=store, path="arr", mode="r").metadata.zarr_format == 2 + + +async def test_create_overwrite_array(backend: str, store: Store) -> None: + zarr.create_group(store=store, path="arr") + await create_overwrite_array(array_metadata(), store, "arr", backend=backend) + assert zarr.open_array(store=store, path="arr", mode="r").shape == (8, 8) + + +async def test_read_metadata(backend: str, store: Store) -> None: + await create_new_array(array_metadata(), store, "arr", backend=backend) + observed = await read_metadata(store, "arr", backend=backend) + raw = await store.get("arr/zarr.json", prototype=default_buffer_prototype()) + assert raw is not None + assert observed == json.loads(raw.to_bytes()) + + +async def test_read_metadata_missing(backend: str, store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await read_metadata(store, "nope", backend=backend) + + +async def test_delete_node(backend: str, store: Store) -> None: + arr = zarr.create_array(store=store, name="doomed", shape=(4,), chunks=(2,), dtype="uint8") + arr[:] = 1 + await delete_node(store, "doomed", backend=backend) + assert not await store.exists("doomed/zarr.json") + assert not await store.exists("doomed/c/0") + + +async def test_delete_node_missing(backend: str, store: Store) -> None: + with pytest.raises(NodeNotFoundError): + await delete_node(store, "nope", backend=backend) + + +async def test_list_children(backend: str, store: Store) -> None: + root = zarr.create_group(store=store) + root.create_group("sub_group", attributes={"kind": "group"}) + root.create_array("sub_array", shape=(4,), chunks=(2,), dtype="uint8") + by_path = dict(await list_children(store, "", backend=backend)) + assert set(by_path) == {"sub_group", "sub_array"} + assert by_path["sub_group"]["node_type"] == "group" + assert by_path["sub_array"]["node_type"] == "array" + assert not any(p.startswith("/") for p in by_path) + + +# --- chunk I/O --- + + +@pytest.mark.parametrize("dtype", ["uint8", "int32", "float64", "u2"]) +async def test_read_chunk_differential(backend: str, store: Store, dtype: str) -> None: + data, meta = filled(store, dtype=dtype) + observed = await read_chunk(meta, store, "a", (1, 0), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 0:4]) + + +@pytest.mark.parametrize( + "compressors", [None, (GzipCodec(),), (ZstdCodec(),), (BloscCodec(cname="lz4"),)] +) +async def test_read_chunk_codecs(backend: str, store: Store, compressors: Any) -> None: + data, meta = filled(store, compressors=compressors) + observed = await read_chunk(meta, store, "a", (0, 1), backend=backend) + np.testing.assert_array_equal(observed, data[0:4, 4:8]) + + +async def test_read_chunk_v2(backend: str, store: Store) -> None: + data, meta = filled(store, dtype=" None: + data, meta = filled(store, dtype="uint16", zarr_format=2, order="F") + observed = await read_chunk(meta, store, "a", (1, 1), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_read_chunk_sharding(backend: str, store: Store) -> None: + data, meta = filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await read_chunk(meta, store, "a", (1, 1), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 4:8]) + + +async def test_read_chunk_missing_is_fill(backend: str, store: Store) -> None: + arr = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 + ) + meta = dict(arr.metadata.to_dict()) + observed = await read_chunk(meta, store, "a", (0, 0), backend=backend) + np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) + + +async def test_read_chunk_metadata_view(backend: str, store: Store) -> None: + data, meta = filled(store, dtype="uint16", compressors=None) + view = copy.deepcopy(meta) + view["data_type"] = "uint8" + view["shape"] = [8, 16] + view["chunk_grid"]["configuration"]["chunk_shape"] = [4, 8] + observed = await read_chunk(view, store, "a", (1, 0), backend=backend) + np.testing.assert_array_equal(observed, data[4:8, 0:4].view("uint8")) + + +async def test_read_chunk_readonly(backend: str, store: Store) -> None: + _, meta = filled(store) + observed = await read_chunk(meta, store, "a", (0, 0), backend=backend) + assert not observed.flags.writeable + + +async def test_write_chunk_differential(backend: str, store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a", backend=backend) + value = np.arange(16, dtype="uint16").reshape(4, 4) + await write_chunk(meta, store, "a", (0, 1), value, backend=backend) + np.testing.assert_array_equal(zarr.open_array(store=store, path="a", mode="r")[0:4, 4:8], value) + + +async def test_write_chunk_shape_mismatch(backend: str, store: Store) -> None: + meta = array_metadata() + await create_new_array(meta, store, "a", backend=backend) + with pytest.raises(ValueError, match="chunk shape"): + await write_chunk( + meta, store, "a", (0, 0), np.zeros((2, 2), dtype="uint16"), backend=backend + ) + + +async def test_delete_chunk(backend: str, store: Store) -> None: + _data, meta = filled(store) + assert await store.exists("a/c/0/0") + await delete_chunk(meta, store, "a", (0, 0), backend=backend) + assert not await store.exists("a/c/0/0") + + +async def test_read_encoded_chunk_matches_store(backend: str, store: Store) -> None: + _, meta = filled(store) + raw = await read_encoded_chunk(meta, store, "a", (0, 0), backend=backend) + expected = await store.get("a/c/0/0", prototype=default_buffer_prototype()) + assert expected is not None + assert raw == expected.to_bytes() + + +async def test_read_encoded_chunk_missing_is_none(backend: str, store: Store) -> None: + arr = zarr.create_array(store=store, name="e", shape=(8, 8), chunks=(4, 4), dtype="uint16") + meta = dict(arr.metadata.to_dict()) + assert await read_encoded_chunk(meta, store, "e", (0, 0), backend=backend) is None + + +# --- region I/O --- + +SELECTIONS: list[Any] = [ + (slice(None), slice(None)), + (slice(2, 7), slice(1, 5)), + (slice(None), 3), + (5, slice(None)), + (3, 4), + (slice(1, 8, 2), slice(None)), + (slice(None), slice(6, 1, -2)), + (slice(-3, None), slice(None, -1)), + ..., + (..., slice(2, 4)), + (slice(0, 0), slice(None)), + (slice(2, 6),), +] + + +@pytest.mark.parametrize("sel", SELECTIONS) +async def test_read_region_differential(backend: str, store: Store, sel: Any) -> None: + data, meta = filled(store) + observed = await read_region(meta, store, "a", sel, backend=backend) + np.testing.assert_array_equal(observed, data[sel]) + + +async def test_read_region_sharding(backend: str, store: Store) -> None: + data, meta = filled(store, chunks=(2, 2), shards=(4, 4)) + observed = await read_region(meta, store, "a", (slice(1, 7), slice(3, 8)), backend=backend) + np.testing.assert_array_equal(observed, data[1:7, 3:8]) + + +async def test_read_region_too_many_indices(backend: str, store: Store) -> None: + _, meta = filled(store) + with pytest.raises(IndexError, match="too many indices"): + await read_region(meta, store, "a", (0, 0, 0), backend=backend) + + +async def test_read_region_fancy_rejected(backend: str, store: Store) -> None: + _, meta = filled(store) + with pytest.raises(TypeError, match="only integers, slices"): + await read_region(meta, store, "a", ([0, 1], slice(None)), backend=backend) # type: ignore[arg-type] From 9dd680abdb56095a074e3b55fefd7db11090da9c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 17:35:23 +0200 Subject: [PATCH 36/75] refactor: share parse_array_metadata; document bytes contract; OOB test Co-Authored-By: Claude Fable 5 --- src/zarr/crud/_api.py | 19 +++---------------- src/zarr/crud/_backend.py | 4 ++++ src/zarr/crud/_common.py | 21 +++++++++++++++++++++ src/zarr/crud/_reference.py | 21 ++++++--------------- tests/crud/test_crud.py | 6 ++++++ 5 files changed, 40 insertions(+), 31 deletions(-) create mode 100644 src/zarr/crud/_common.py diff --git a/src/zarr/crud/_api.py b/src/zarr/crud/_api.py index 5fdb8fe1d2..91aeef5007 100644 --- a/src/zarr/crud/_api.py +++ b/src/zarr/crud/_api.py @@ -9,6 +9,7 @@ import numpy as np from zarr.core.buffer.core import default_buffer_prototype +from zarr.crud._common import parse_array_metadata from zarr.crud._registry import get_backend if TYPE_CHECKING: @@ -18,8 +19,6 @@ from zarr.abc.store import Store from zarr.core.common import JSON - from zarr.core.metadata.v2 import ArrayV2Metadata - from zarr.core.metadata.v3 import ArrayV3Metadata from zarr.crud._backend import CrudBackend @@ -42,18 +41,6 @@ def _resolve_backend(backend: CrudBackend | str | None) -> CrudBackend: return backend -def _parse_array_metadata( - metadata: Mapping[str, JSON], -) -> ArrayV3Metadata | ArrayV2Metadata: - from zarr.core.metadata.v2 import ArrayV2Metadata - from zarr.core.metadata.v3 import ArrayV3Metadata - - data = dict(metadata) - if data.get("zarr_format") == 3: - return ArrayV3Metadata.from_dict(data) - return ArrayV2Metadata.from_dict(data) - - def _chunk_dtype_and_shape( metadata: Mapping[str, JSON], ) -> tuple[np.dtype[Any], tuple[int, ...]]: @@ -64,7 +51,7 @@ def _chunk_dtype_and_shape( """ from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) if isinstance(meta_obj, ArrayV3Metadata): grid = meta_obj.chunk_grid if not isinstance(grid, RegularChunkGridMetadata): @@ -90,7 +77,7 @@ def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: def _chunk_key(metadata: Mapping[str, JSON], path: str, coords: tuple[int, ...]) -> str: - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) rel = meta_obj.encode_chunk_key(coords) p = path.strip("/") return f"{p}/{rel}" if p else rel diff --git a/src/zarr/crud/_backend.py b/src/zarr/crud/_backend.py index 79fa1f49df..638dacf6f6 100644 --- a/src/zarr/crud/_backend.py +++ b/src/zarr/crud/_backend.py @@ -29,6 +29,10 @@ class CrudBackend(Protocol): Note: because this protocol is `runtime_checkable`, `isinstance` checks only verify that the method names exist, not their signatures or that they are async. Static type checking (mypy) is the authoritative conformance check. + + `read_chunk` and `read_subset` must return immutable `bytes` (not + `bytearray`): the facade wraps them with `numpy.frombuffer`, which yields a + read-only array only for immutable buffers. """ async def create_array( diff --git a/src/zarr/crud/_common.py b/src/zarr/crud/_common.py new file mode 100644 index 0000000000..4837edfa03 --- /dev/null +++ b/src/zarr/crud/_common.py @@ -0,0 +1,21 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from zarr.core.metadata.v2 import ArrayV2Metadata +from zarr.core.metadata.v3 import ArrayV3Metadata + +if TYPE_CHECKING: + from collections.abc import Mapping + + from zarr.core.common import JSON + + +def parse_array_metadata( + metadata: Mapping[str, JSON], +) -> ArrayV3Metadata | ArrayV2Metadata: + """Parse a metadata document into a v2 or v3 array metadata object.""" + data = dict(metadata) + if data.get("zarr_format") == 3: + return ArrayV3Metadata.from_dict(data) + return ArrayV2Metadata.from_dict(data) diff --git a/src/zarr/crud/_reference.py b/src/zarr/crud/_reference.py index e8b76571bd..2b48186e56 100644 --- a/src/zarr/crud/_reference.py +++ b/src/zarr/crud/_reference.py @@ -13,6 +13,7 @@ from zarr.core.metadata.v2 import ArrayV2Metadata from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata from zarr.crud._backend import NodeExistsError +from zarr.crud._common import parse_array_metadata from zarr.errors import NodeNotFoundError from zarr.storage._common import StorePath @@ -23,16 +24,6 @@ from zarr.core.common import JSON -def _parse_array_metadata( - metadata: Mapping[str, JSON], -) -> ArrayV3Metadata | ArrayV2Metadata: - """Parse a metadata document into a v2 or v3 array metadata object.""" - data = dict(metadata) - if data.get("zarr_format") == 3: - return ArrayV3Metadata.from_dict(data) - return ArrayV2Metadata.from_dict(data) - - def _native_dtype(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> np.dtype[Any]: """Numpy dtype in native byte order (zarrs and the facade assume native).""" return meta_obj.dtype.to_native_dtype().newbyteorder("=") @@ -75,7 +66,7 @@ class ReferenceBackend: async def create_array( self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool ) -> None: - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) async def create_group( @@ -118,7 +109,7 @@ async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: async def read_chunk( self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] ) -> bytes: - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) shape = _chunk_shape(meta_obj) np_dtype = _native_dtype(meta_obj) sp = StorePath(store, path.strip("/")) @@ -145,7 +136,7 @@ async def read_subset( start: Sequence[int], shape: Sequence[int], ) -> bytes: - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) np_dtype = _native_dtype(meta_obj) async_arr = AsyncArray(metadata=meta_obj, store_path=StorePath(store, path.strip("/"))) selection = tuple(slice(s, s + length) for s, length in zip(start, shape, strict=True)) @@ -160,7 +151,7 @@ async def write_chunk( coords: tuple[int, ...], data: bytes, ) -> None: - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) shape = _chunk_shape(meta_obj) np_dtype = _native_dtype(meta_obj) sp = StorePath(store, path.strip("/")) @@ -178,7 +169,7 @@ async def write_chunk( async def delete_chunk( self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] ) -> None: - meta_obj = _parse_array_metadata(metadata) + meta_obj = parse_array_metadata(metadata) sp = StorePath(store, path.strip("/")) await (sp / meta_obj.encode_chunk_key(coords)).delete() diff --git a/tests/crud/test_crud.py b/tests/crud/test_crud.py index c301725e22..2b406c0a5f 100644 --- a/tests/crud/test_crud.py +++ b/tests/crud/test_crud.py @@ -252,3 +252,9 @@ async def test_read_region_fancy_rejected(backend: str, store: Store) -> None: _, meta = filled(store) with pytest.raises(TypeError, match="only integers, slices"): await read_region(meta, store, "a", ([0, 1], slice(None)), backend=backend) # type: ignore[arg-type] + + +async def test_read_region_out_of_bounds(backend: str, store: Store) -> None: + _, meta = filled(store) + with pytest.raises(IndexError, match="out of bounds"): + await read_region(meta, store, "a", (8, slice(None)), backend=backend) From 26af3e4a015cf21cb8a4af1b6e344685aaba05fc Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 17:41:50 +0200 Subject: [PATCH 37/75] feat: ZarrsBackend conforms to CrudBackend; zarr.zarrs is now a backend Replaces the old flat-function API in `zarr.zarrs._api` with `ZarrsBackend` (a `CrudBackend` implementation) in `zarr.zarrs._backend`. Importing `zarr.zarrs` now registers the backend under the key `"zarrs"` via `zarr.crud.register_backend`, enabling `backend="zarrs"` in all crud facade functions and activating the previously-skipped zarrs params in `tests/crud/test_crud.py`. Old tests that exercised the removed flat API are deleted; cache tests are migrated to use the `zarr.crud` facade. Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/__init__.py | 50 +---- src/zarr/zarrs/_api.py | 428 ------------------------------------- src/zarr/zarrs/_backend.py | 143 +++++++++++++ tests/zarrs/test_api.py | 13 -- tests/zarrs/test_cache.py | 23 +- tests/zarrs/test_chunk.py | 226 -------------------- tests/zarrs/test_node.py | 143 ------------- 7 files changed, 165 insertions(+), 861 deletions(-) delete mode 100644 src/zarr/zarrs/_api.py create mode 100644 src/zarr/zarrs/_backend.py delete mode 100644 tests/zarrs/test_api.py delete mode 100644 tests/zarrs/test_chunk.py delete mode 100644 tests/zarrs/test_node.py diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py index 1e287b9cfa..bff68ade62 100644 --- a/src/zarr/zarrs/__init__.py +++ b/src/zarr/zarrs/__init__.py @@ -1,13 +1,11 @@ """ -Low-level functional API for zarr hierarchies, backed by the Rust +The zarrs CRUD backend for `zarr.crud`, backed by the Rust [`zarrs`](https://zarrs.dev) crate. -This subpackage is experimental. It requires the `zarrs-bindings` package -(in-repo Rust crate; install for development with `uv sync --group zarrs`). - -All array routines take an explicit metadata document (a `dict` matching the -`zarr.json` / `.zarray` document) rather than reading metadata from the store, -which makes read-only and virtual views possible. +Importing this module registers the `"zarrs"` backend. Requires the +`zarrs-bindings` extension (in-repo Rust crate; `uv sync --group zarrs`). Select +it with `zarr.config.set({"crud.backend": "zarrs"})` or per call via +`backend="zarrs"`. """ try: @@ -18,39 +16,11 @@ "It is built from the zarr-python repository: run `uv sync --group zarrs`." ) from e +from zarr.crud import register_backend +from zarr.zarrs._backend import ZarrsBackend + __version__: str = _zarrs_bindings.version() -from zarr.zarrs._api import ( - NodeExistsError, - ZarrsOptions, - create_new_array, - create_new_group, - create_overwrite_array, - create_overwrite_group, - decode_chunk, - decode_region, - delete_node, - encode_chunk, - erase_chunk, - list_children, - read_encoded_chunk, - read_metadata, -) +register_backend("zarrs", ZarrsBackend()) -__all__ = [ - "NodeExistsError", - "ZarrsOptions", - "__version__", - "create_new_array", - "create_new_group", - "create_overwrite_array", - "create_overwrite_group", - "decode_chunk", - "decode_region", - "delete_node", - "encode_chunk", - "erase_chunk", - "list_children", - "read_encoded_chunk", - "read_metadata", -] +__all__ = ["ZarrsBackend", "__version__"] diff --git a/src/zarr/zarrs/_api.py b/src/zarr/zarrs/_api.py deleted file mode 100644 index df0804f719..0000000000 --- a/src/zarr/zarrs/_api.py +++ /dev/null @@ -1,428 +0,0 @@ -from __future__ import annotations - -import asyncio -import json -import operator -import types -from collections.abc import Sequence -from contextlib import contextmanager -from dataclasses import dataclass -from typing import TYPE_CHECKING, Any, cast - -import _zarrs_bindings as _zb -import numpy as np - -from zarr.errors import NodeNotFoundError -from zarr.zarrs._bridge import resolve_store - -if TYPE_CHECKING: - from collections.abc import Iterator, Mapping - - import numpy.typing as npt - - from zarr.abc.store import Store - from zarr.core.common import JSON - -NodeExistsError = _zb.NodeExistsError -"""Raised by `create_new_*` when a node already exists at the target path.""" - - -@dataclass(frozen=True, slots=True) -class ZarrsOptions: - """Options for zarrs-backed operations. - - Currently empty: fields (concurrency limits, checksum validation) arrive in - a later phase. Accepting it now keeps signatures stable. - """ - - -BasicIndex = int | slice | types.EllipsisType -BasicSelection = BasicIndex | tuple[BasicIndex, ...] - - -def _node_path(path: str) -> str: - """Convert a zarr-python node path (`""`, `"foo/bar"`) to a zarrs node path - (`"/"`, `"/foo/bar"`).""" - return f"/{path.strip('/')}" - - -@contextmanager -def _translate_errors() -> Iterator[None]: - try: - yield - except _zb.NodeNotFoundError as err: - raise NodeNotFoundError(str(err)) from err - - -async def create_new_group( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> None: - """Create a group at `path` from a group metadata document. - - Raises `NodeExistsError` if any node already exists at `path`. - Creation is not atomic with respect to concurrent writers: a concurrent - creation at the same path can race the existence check. - """ - with _translate_errors(): - await asyncio.to_thread( - _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), False - ) - - -async def create_overwrite_group( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> None: - """Create a group at `path`, deleting any existing node (and its children) first. - - Creation is not atomic with respect to concurrent writers: a concurrent - creation at the same path can race the existence check. - """ - with _translate_errors(): - await asyncio.to_thread( - _zb.create_group, resolve_store(store), _node_path(path), json.dumps(metadata), True - ) - - -async def create_new_array( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> None: - """Create an array at `path` from a v2 or v3 array metadata document. - - Raises `NodeExistsError` if any node already exists at `path`. Creation is - not atomic with respect to concurrent writers: a concurrent creation at the - same path can race the existence check. - """ - with _translate_errors(): - await asyncio.to_thread( - _zb.create_array, resolve_store(store), _node_path(path), json.dumps(metadata), False - ) - - -async def create_overwrite_array( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> None: - """Create an array at `path`, deleting any existing node (and its children) - first. The delete-then-create sequence is not atomic with respect to - concurrent writers. - """ - with _translate_errors(): - await asyncio.to_thread( - _zb.create_array, resolve_store(store), _node_path(path), json.dumps(metadata), True - ) - - -async def read_metadata( - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> dict[str, JSON]: - """Read the metadata document of the array or group at `path`. - - Raises `zarr.errors.NodeNotFoundError` if no node exists there. - """ - with _translate_errors(): - raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) - return cast("dict[str, JSON]", json.loads(raw)) - - -async def delete_node( - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> None: - """Delete the node at `path`, including all keys and child nodes under it. - - Raises `zarr.errors.NodeNotFoundError` if no node exists there. Deleting - the root node (`path=""`) clears the entire store. - """ - with _translate_errors(): - await asyncio.to_thread(_zb.delete_node, resolve_store(store), _node_path(path)) - - -async def list_children( - store: Store, - path: str, - *, - options: ZarrsOptions | None = None, -) -> list[tuple[str, dict[str, JSON]]]: - """List the direct children of the group at `path` as - `(path, metadata_document)` pairs. Paths are store-relative (no leading - `/`). - - Raises `zarr.errors.NodeNotFoundError` if no *group* exists at `path` -- - including when `path` holds an array. - """ - with _translate_errors(): - raw: list[tuple[str, str]] = await asyncio.to_thread( - _zb.list_children, resolve_store(store), _node_path(path) - ) - return [ - (child_path.lstrip("/"), cast("dict[str, JSON]", json.loads(doc))) - for child_path, doc in raw - ] - - -def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: - """Resolve the array shape from a metadata document.""" - shape = metadata.get("shape") - if not isinstance(shape, Sequence) or isinstance(shape, str): - raise TypeError("metadata document has no valid 'shape'") - result: list[int] = [] - for s in shape: - if not isinstance(s, (int, float)): - raise TypeError(f"shape element {s!r} is not a number") - if isinstance(s, float) and not s.is_integer(): - raise TypeError(f"shape element {s!r} is not an integer") - result.append(int(s)) - return tuple(result) - - -def _normalize_selection( - selection: BasicSelection, shape: tuple[int, ...] -) -> tuple[list[int], list[int], tuple[slice | int, ...]]: - """Normalize a numpy-style basic-indexing selection against `shape`. - - Returns `(start, bounding_shape, post_index)`: the step-1 bounding box to - fetch (per-dimension start and length), and the numpy index to apply to - the fetched block to produce the final result (strides, reversals, and - integer-axis removal). Only integers, slices, and `Ellipsis` are - supported; fancy indexing raises `TypeError`. - """ - sel_tuple = selection if isinstance(selection, tuple) else (selection,) - - n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) - if n_ellipsis > 1: - raise IndexError("an index can only have a single ellipsis ('...')") - if n_ellipsis == 1: - i = sel_tuple.index(Ellipsis) - n_fill = len(shape) - (len(sel_tuple) - 1) - if n_fill < 0: - raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") - sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] - if len(sel_tuple) > len(shape): - raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") - sel_tuple = sel_tuple + (slice(None),) * (len(shape) - len(sel_tuple)) - - starts: list[int] = [] - lengths: list[int] = [] - post: list[slice | int] = [] - for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): - if isinstance(sel, slice): - start, stop, step = sel.indices(size) - n = len(range(start, stop, step)) - if n == 0: - starts.append(0) - lengths.append(0) - post.append(slice(None)) - elif step > 0: - last = start + (n - 1) * step - starts.append(start) - lengths.append(last - start + 1) - post.append(slice(None, None, step)) - else: - # descending: bounding box is [last, start], ascending in store - # order; slice(None, None, step) over the block starts at its - # final element (global `start`) and lands exactly on index 0 - # (global `last`) because the block length is (n-1)*|step| + 1. - last = start + (n - 1) * step - starts.append(last) - lengths.append(start - last + 1) - post.append(slice(None, None, step)) - else: - assert not isinstance(sel, types.EllipsisType), "Ellipsis already expanded above" - try: - idx = operator.index(sel) - except TypeError: - raise TypeError( - "unsupported selection element " - f"{sel!r}: only integers, slices, and Ellipsis are supported" - ) from None - if idx < 0: - idx += size - if not 0 <= idx < size: - raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") - starts.append(idx) - lengths.append(1) - post.append(0) - return starts, lengths, tuple(post) - - -def _chunk_dtype_and_shape( - metadata: Mapping[str, JSON], -) -> tuple[np.dtype[Any], tuple[int, ...]]: - """Resolve the numpy dtype and chunk shape from a metadata document, using - zarr-python's own metadata parsing. - - The dtype is coerced to native byte order: zarrs always decodes to (and - encodes from) the native in-memory representation, applying any byte-order - codec itself. - """ - from zarr.core.metadata.v2 import ArrayV2Metadata - from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata - - if metadata.get("zarr_format") == 3: - meta3 = ArrayV3Metadata.from_dict(dict(metadata)) - grid = meta3.chunk_grid - if not isinstance(grid, RegularChunkGridMetadata): - raise NotImplementedError("only regular chunk grids are supported") - return meta3.data_type.to_native_dtype().newbyteorder("="), grid.chunk_shape - meta2 = ArrayV2Metadata.from_dict(dict(metadata)) - return meta2.dtype.to_native_dtype().newbyteorder("="), meta2.chunks - - -async def decode_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - *, - selection: tuple[slice | int, ...] | None = None, - options: ZarrsOptions | None = None, -) -> np.ndarray[Any, np.dtype[Any]]: - """Read and decode the chunk at `chunk_coords` of the array described by - `metadata`, located at `path` in `store`. - - The metadata document is authoritative: it is not read from the store. - Missing chunks decode to the fill value. `selection` (a chunk-relative - subset) is not implemented yet. - - The returned array is a read-only, zero-copy view over the decoded bytes; - call `.copy()` if you need a writable array. - """ - if selection is not None: - raise NotImplementedError("chunk subset selection is not implemented yet") - raw = await asyncio.to_thread( - _zb.retrieve_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(chunk_coords), - ) - dtype, chunk_shape = _chunk_dtype_and_shape(metadata) - return np.frombuffer(raw, dtype=dtype).reshape(chunk_shape) - - -async def read_encoded_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - *, - options: ZarrsOptions | None = None, -) -> bytes | None: - """Read the raw, still-encoded bytes of the chunk at `chunk_coords`, or - `None` if the chunk does not exist. No codecs are applied.""" - result: bytes | None = await asyncio.to_thread( - _zb.retrieve_encoded_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(chunk_coords), - ) - return result - - -async def encode_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - value: npt.ArrayLike, - *, - options: ZarrsOptions | None = None, -) -> None: - """Encode `value` with the codecs in `metadata` and store it as the chunk - at `chunk_coords`. `value` must match the chunk shape exactly.""" - dtype, chunk_shape = _chunk_dtype_and_shape(metadata) - arr = np.ascontiguousarray(np.asarray(value, dtype=dtype)) - if arr.shape != chunk_shape: - raise ValueError(f"value shape {arr.shape} does not match chunk shape {chunk_shape}") - await asyncio.to_thread( - _zb.store_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(chunk_coords), - arr.tobytes(), - ) - - -async def erase_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - *, - options: ZarrsOptions | None = None, -) -> None: - """Delete the chunk at `chunk_coords`. Deleting a missing chunk is a no-op.""" - await asyncio.to_thread( - _zb.erase_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(chunk_coords), - ) - - -async def decode_region( - metadata: Mapping[str, JSON], - store: Store, - path: str, - selection: BasicSelection, - *, - options: ZarrsOptions | None = None, -) -> np.ndarray[Any, np.dtype[Any]]: - """Read and decode the region of the array described by `metadata` given - by a numpy-style basic-indexing `selection` (integers, slices including - steps, `Ellipsis`). - - The metadata document is authoritative: it is not read from the store. - One zarrs call fetches the step-1 bounding box of the selection (decoding - all overlapping chunks, in parallel for multi-chunk regions); strides, - reversals, and integer-axis removal are applied as numpy views on the - result. Missing chunks decode to the fill value. Fancy indexing (integer - or boolean arrays) is not supported and raises `TypeError`. The returned - array is a read-only view; call `.copy()` if you need a writable array. - - Note: zarrs fetches the step-1 bounding box of the selection. A selection - like `slice(0, N, step)` reads `O(N)` bytes from the store even though only - `O(N / step)` are returned; for sparse selections over large arrays, prefer - reading per-chunk with `decode_chunk`. - """ - dtype, _ = _chunk_dtype_and_shape(metadata) - shape = _array_shape(metadata) - starts, lengths, post_index = _normalize_selection(selection, shape) - if 0 in lengths: - block = np.empty(lengths, dtype=dtype) - block.flags.writeable = False - else: - raw = await asyncio.to_thread( - _zb.retrieve_array_subset, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - starts, - lengths, - ) - block = np.frombuffer(raw, dtype=dtype).reshape(lengths) - result: np.ndarray[Any, np.dtype[Any]] = block[post_index] - return result diff --git a/src/zarr/zarrs/_backend.py b/src/zarr/zarrs/_backend.py new file mode 100644 index 0000000000..196cc10af0 --- /dev/null +++ b/src/zarr/zarrs/_backend.py @@ -0,0 +1,143 @@ +from __future__ import annotations + +import asyncio +import json +from contextlib import contextmanager +from typing import TYPE_CHECKING, cast + +import _zarrs_bindings as _zb + +from zarr.crud import NodeExistsError +from zarr.errors import NodeNotFoundError +from zarr.zarrs._bridge import resolve_store + +if TYPE_CHECKING: + from collections.abc import Iterator, Mapping, Sequence + + from zarr.abc.store import Store + from zarr.core.common import JSON + + +def _node_path(path: str) -> str: + """Convert a zarr path (`""`, `"foo/bar"`) to a zarrs node path (`"/"`, + `"/foo/bar"`).""" + return f"/{path.strip('/')}" + + +@contextmanager +def _translate_errors() -> Iterator[None]: + try: + yield + except _zb.NodeNotFoundError as err: + raise NodeNotFoundError(str(err)) from err + except _zb.NodeExistsError as err: + raise NodeExistsError(str(err)) from err + + +class ZarrsBackend: + """CRUD backend backed by the Rust `zarrs` crate via `_zarrs_bindings`. + + Owns the zarrs-specific plumbing: JSON-serializing the metadata document, + the `/`-prefixed node-path form, store resolution, offloading the blocking + Rust calls to a worker thread, and translating binding exceptions to the + canonical `zarr.crud` / `zarr.errors` types. + """ + + async def create_array( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + with _translate_errors(): + await asyncio.to_thread( + _zb.create_array, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + overwrite, + ) + + async def create_group( + self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool + ) -> None: + with _translate_errors(): + await asyncio.to_thread( + _zb.create_group, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + overwrite, + ) + + async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: + with _translate_errors(): + raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) + return cast("dict[str, JSON]", json.loads(raw)) + + async def read_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> bytes: + return await asyncio.to_thread( + _zb.retrieve_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(coords), + ) + + async def read_subset( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + start: Sequence[int], + shape: Sequence[int], + ) -> bytes: + return await asyncio.to_thread( + _zb.retrieve_array_subset, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(start), + list(shape), + ) + + async def write_chunk( + self, + store: Store, + path: str, + metadata: Mapping[str, JSON], + coords: tuple[int, ...], + data: bytes, + ) -> None: + await asyncio.to_thread( + _zb.store_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(coords), + data, + ) + + async def delete_chunk( + self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] + ) -> None: + await asyncio.to_thread( + _zb.erase_chunk, + resolve_store(store), + _node_path(path), + json.dumps(metadata), + list(coords), + ) + + async def delete_node(self, store: Store, path: str) -> None: + with _translate_errors(): + await asyncio.to_thread(_zb.delete_node, resolve_store(store), _node_path(path)) + + async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: + with _translate_errors(): + raw: list[tuple[str, str]] = await asyncio.to_thread( + _zb.list_children, resolve_store(store), _node_path(path) + ) + return [ + (child_path.lstrip("/"), cast("dict[str, JSON]", json.loads(doc))) + for child_path, doc in raw + ] diff --git a/tests/zarrs/test_api.py b/tests/zarrs/test_api.py deleted file mode 100644 index 1a3e9005e2..0000000000 --- a/tests/zarrs/test_api.py +++ /dev/null @@ -1,13 +0,0 @@ -from __future__ import annotations - -import pytest - -pytest.importorskip( - "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError -) - - -def test_import() -> None: - import zarr.zarrs - - assert isinstance(zarr.zarrs.__version__, str) diff --git a/tests/zarrs/test_cache.py b/tests/zarrs/test_cache.py index 41e3e24f29..9af72555d8 100644 --- a/tests/zarrs/test_cache.py +++ b/tests/zarrs/test_cache.py @@ -12,8 +12,9 @@ import _zarrs_bindings as zb import zarr +import zarr.zarrs # registers the "zarrs" CrudBackend +from zarr.crud import read_chunk, write_chunk from zarr.storage import LocalStore, MemoryStore -from zarr.zarrs import decode_chunk, encode_chunk if TYPE_CHECKING: from pathlib import Path @@ -34,17 +35,17 @@ async def test_localstore_populates_cache(tmp_path: Path) -> None: store = await LocalStore.open(root=tmp_path / "s") meta = _meta(store) assert zb.array_cache_len() == 0 - await decode_chunk(meta, store, "a", (0, 0)) + await read_chunk(meta, store, "a", (0, 0), backend="zarrs") assert zb.array_cache_len() == 1 # second op on the SAME array reuses the entry, does not grow the cache - await decode_chunk(meta, store, "a", (1, 1)) + await read_chunk(meta, store, "a", (1, 1), backend="zarrs") assert zb.array_cache_len() == 1 async def test_memorystore_is_not_cached() -> None: store = MemoryStore() meta = _meta(store) - await decode_chunk(meta, store, "a", (0, 0)) + await read_chunk(meta, store, "a", (0, 0), backend="zarrs") assert zb.array_cache_len() == 0 @@ -52,8 +53,8 @@ async def test_distinct_metadata_distinct_entries(tmp_path: Path) -> None: store = await LocalStore.open(root=tmp_path / "s") meta_a = _meta(store, "a") meta_b = _meta(store, "b") - await decode_chunk(meta_a, store, "a", (0, 0)) - await decode_chunk(meta_b, store, "b", (0, 0)) + await read_chunk(meta_a, store, "a", (0, 0), backend="zarrs") + await read_chunk(meta_b, store, "b", (0, 0), backend="zarrs") assert zb.array_cache_len() == 2 @@ -67,8 +68,8 @@ async def test_cache_keyed_on_root_not_just_metadata(tmp_path: Path) -> None: a2 = zarr.create_array(store=s2, name="a", shape=(4, 4), chunks=(4, 4), dtype="uint16") a2[:, :] = 2 meta = dict(a1.metadata.to_dict()) # identical metadata document - out1 = await decode_chunk(meta, s1, "a", (0, 0)) - out2 = await decode_chunk(meta, s2, "a", (0, 0)) + out1 = await read_chunk(meta, s1, "a", (0, 0), backend="zarrs") + out2 = await read_chunk(meta, s2, "a", (0, 0), backend="zarrs") np.testing.assert_array_equal(out1, np.full((4, 4), 1, dtype="uint16")) np.testing.assert_array_equal(out2, np.full((4, 4), 2, dtype="uint16")) assert zb.array_cache_len() == 2 @@ -79,8 +80,8 @@ async def test_cache_reflects_writes_through_store(tmp_path: Path) -> None: # a subsequent read (proves the cache does not stale-cache chunk data) store = await LocalStore.open(root=tmp_path / "s") meta = _meta(store) - await decode_chunk(meta, store, "a", (0, 0)) # caches the Array + await read_chunk(meta, store, "a", (0, 0), backend="zarrs") # caches the Array new = np.full((4, 4), 99, dtype="uint16") - await encode_chunk(meta, store, "a", (0, 0), new) # write via (cached) Array - out = await decode_chunk(meta, store, "a", (0, 0)) + await write_chunk(meta, store, "a", (0, 0), new, backend="zarrs") # write via (cached) Array + out = await read_chunk(meta, store, "a", (0, 0), backend="zarrs") np.testing.assert_array_equal(out, new) diff --git a/tests/zarrs/test_chunk.py b/tests/zarrs/test_chunk.py deleted file mode 100644 index 6b4964413c..0000000000 --- a/tests/zarrs/test_chunk.py +++ /dev/null @@ -1,226 +0,0 @@ -from __future__ import annotations - -import copy -from typing import TYPE_CHECKING, Any - -import numpy as np -import pytest - -pytest.importorskip( - "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError -) - -import zarr -from tests.zarrs.conftest import array_metadata -from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec -from zarr.core.buffer.core import default_buffer_prototype -from zarr.zarrs import ( - create_new_array, - decode_chunk, - decode_region, - encode_chunk, - erase_chunk, - read_encoded_chunk, -) - -if TYPE_CHECKING: - from zarr.abc.store import Store - - -def _filled(store: Store, **kwargs: Any) -> tuple[np.ndarray[Any, np.dtype[Any]], dict[str, Any]]: - """Create an 8x8 array named 'a' via zarr-python, fill it with a ramp, and - return (data, metadata_document).""" - params: dict[str, Any] = {"shape": (8, 8), "chunks": (4, 4), "dtype": "uint16"} | kwargs - arr = zarr.create_array(store=store, name="a", **params) - data = np.arange(64, dtype=params["dtype"]).reshape(8, 8) - arr[:, :] = data - doc = dict(arr.metadata.to_dict()) - if params.get("zarr_format") == 2: - # v2 attributes live in .zattrs, not in the .zarray document - doc.pop("attributes", None) - return data, doc - - -@pytest.mark.parametrize("dtype", ["uint8", "int32", "float64"]) -async def test_decode_chunk_differential(store: Store, dtype: str) -> None: - data, meta = _filled(store, dtype=dtype) - observed = await decode_chunk(meta, store, "a", (1, 0)) - np.testing.assert_array_equal(observed, data[4:8, 0:4]) - - -@pytest.mark.parametrize( - "compressors", [None, (GzipCodec(),), (ZstdCodec(),), (BloscCodec(cname="lz4"),)] -) -async def test_decode_chunk_codecs(store: Store, compressors: Any) -> None: - data, meta = _filled(store, compressors=compressors) - observed = await decode_chunk(meta, store, "a", (0, 1)) - np.testing.assert_array_equal(observed, data[0:4, 4:8]) - - -async def test_decode_chunk_v2(store: Store) -> None: - data, meta = _filled(store, zarr_format=2) - observed = await decode_chunk(meta, store, "a", (1, 1)) - np.testing.assert_array_equal(observed, data[4:8, 4:8]) - - -async def test_decode_chunk_v2_big_endian(store: Store) -> None: - data, meta = _filled(store, dtype=">u2", zarr_format=2) - observed = await decode_chunk(meta, store, "a", (1, 1)) - np.testing.assert_array_equal(observed, data[4:8, 4:8]) - - -async def test_encode_chunk_v2_big_endian(store: Store) -> None: - meta = array_metadata(dtype=">u2", zarr_format=2) - await create_new_array(meta, store, "a") - value = np.arange(16, dtype="uint16").reshape(4, 4) - await encode_chunk(meta, store, "a", (0, 1), value) - arr = zarr.open_array(store=store, path="a", mode="r") - np.testing.assert_array_equal(arr[0:4, 4:8], value) - - -async def test_decode_chunk_readonly(store: Store) -> None: - _, meta = _filled(store) - observed = await decode_chunk(meta, store, "a", (0, 0)) - assert not observed.flags.writeable - - -async def test_decode_chunk_sharding(store: Store) -> None: - # with sharding, the metadata chunk grid is the shard grid - data, meta = _filled(store, chunks=(2, 2), shards=(4, 4)) - observed = await decode_chunk(meta, store, "a", (1, 1)) - np.testing.assert_array_equal(observed, data[4:8, 4:8]) - - -async def test_decode_chunk_missing_returns_fill_value(store: Store) -> None: - arr = zarr.create_array( - store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 - ) - meta = dict(arr.metadata.to_dict()) - observed = await decode_chunk(meta, store, "a", (0, 0)) - np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) - - -async def test_decode_chunk_selection_not_implemented(store: Store) -> None: - _, meta = _filled(store) - with pytest.raises(NotImplementedError): - await decode_chunk(meta, store, "a", (0, 0), selection=(slice(0, 2), slice(0, 2))) - - -async def test_decode_chunk_metadata_view(store: Store) -> None: - # the read-only-view case: decode with a metadata document the store never saw - data, meta = _filled(store, dtype="uint16", compressors=None) - view = copy.deepcopy(meta) - view["data_type"] = "uint8" - view["shape"] = [8, 16] - view["chunk_grid"]["configuration"]["chunk_shape"] = [4, 8] - observed = await decode_chunk(view, store, "a", (1, 0)) - np.testing.assert_array_equal(observed, data[4:8, 0:4].view("uint8")) - - -async def test_encode_chunk_differential(store: Store) -> None: - meta = array_metadata() - await create_new_array(meta, store, "a") - value = np.arange(16, dtype="uint16").reshape(4, 4) - await encode_chunk(meta, store, "a", (0, 1), value) - arr = zarr.open_array(store=store, path="a", mode="r") - np.testing.assert_array_equal(arr[0:4, 4:8], value) - - -async def test_encode_chunk_shape_mismatch(store: Store) -> None: - meta = array_metadata() - await create_new_array(meta, store, "a") - with pytest.raises(ValueError, match="chunk shape"): - await encode_chunk(meta, store, "a", (0, 0), np.zeros((2, 2), dtype="uint16")) - - -async def test_read_encoded_chunk_matches_store(store: Store) -> None: - _, meta = _filled(store) - raw = await read_encoded_chunk(meta, store, "a", (0, 0)) - expected = await store.get("a/c/0/0", prototype=default_buffer_prototype()) - assert expected is not None - assert raw == expected.to_bytes() - - -async def test_read_encoded_chunk_missing_returns_none(store: Store) -> None: - arr = zarr.create_array(store=store, name="empty", shape=(8, 8), chunks=(4, 4), dtype="uint16") - meta = dict(arr.metadata.to_dict()) - assert await read_encoded_chunk(meta, store, "empty", (0, 0)) is None - - -async def test_erase_chunk(store: Store) -> None: - _, meta = _filled(store) - assert await store.exists("a/c/0/0") - await erase_chunk(meta, store, "a", (0, 0)) - assert not await store.exists("a/c/0/0") - arr = zarr.open_array(store=store, path="a", mode="r") - np.testing.assert_array_equal(arr[0:4, 0:4], np.zeros((4, 4), dtype="uint16")) - - -SELECTIONS: list[Any] = [ - (slice(None), slice(None)), - (slice(2, 7), slice(1, 5)), # crosses chunk boundaries - (slice(None), 3), - (5, slice(None)), - (3, 4), # fully scalar -> 0-d result - (slice(1, 8, 2), slice(None)), - (slice(None), slice(6, 1, -2)), # negative step - (slice(-3, None), slice(None, -1)), # negative bounds - ..., # Ellipsis alone - (..., slice(2, 4)), - (slice(0, 0), slice(None)), # empty - (slice(2, 6),), # partial selection, missing trailing dims -] - - -@pytest.mark.parametrize("sel", SELECTIONS) -async def test_decode_region_differential(store: Store, sel: Any) -> None: - data, meta = _filled(store) - observed = await decode_region(meta, store, "a", sel) - np.testing.assert_array_equal(observed, data[sel]) - - -async def test_decode_region_sharding(store: Store) -> None: - data, meta = _filled(store, chunks=(2, 2), shards=(4, 4)) - observed = await decode_region(meta, store, "a", (slice(1, 7), slice(3, 8))) - np.testing.assert_array_equal(observed, data[1:7, 3:8]) - - -async def test_decode_region_v2(store: Store) -> None: - data, meta = _filled(store, zarr_format=2) - observed = await decode_region(meta, store, "a", (slice(2, 7), slice(None, None, 3))) - np.testing.assert_array_equal(observed, data[2:7, ::3]) - - -async def test_decode_region_missing_chunks_fill_value(store: Store) -> None: - arr = zarr.create_array( - store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 - ) - meta = dict(arr.metadata.to_dict()) - observed = await decode_region(meta, store, "a", (slice(2, 6), slice(2, 6))) - np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) - - -async def test_decode_region_out_of_bounds(store: Store) -> None: - _, meta = _filled(store) - with pytest.raises(IndexError, match="out of bounds"): - await decode_region(meta, store, "a", (8, slice(None))) - - -async def test_decode_region_too_many_indices(store: Store) -> None: - _, meta = _filled(store) - with pytest.raises(IndexError, match="too many indices"): - await decode_region(meta, store, "a", (0, 0, 0)) - - -async def test_decode_region_fancy_indexing_rejected(store: Store) -> None: - _, meta = _filled(store) - with pytest.raises(TypeError, match="only integers, slices"): - await decode_region(meta, store, "a", ([0, 1], slice(None))) # type: ignore[arg-type] - - -async def test_decode_region_readonly(store: Store) -> None: - _, meta = _filled(store) - observed = await decode_region(meta, store, "a", (slice(0, 4), slice(0, 4))) - assert not observed.flags.writeable - empty = await decode_region(meta, store, "a", (slice(0, 0), slice(None))) - assert not empty.flags.writeable diff --git a/tests/zarrs/test_node.py b/tests/zarrs/test_node.py deleted file mode 100644 index eefb4cf137..0000000000 --- a/tests/zarrs/test_node.py +++ /dev/null @@ -1,143 +0,0 @@ -from __future__ import annotations - -import json -from typing import TYPE_CHECKING, Any - -import numpy as np -import pytest - -pytest.importorskip( - "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError -) - -import zarr -from tests.zarrs.conftest import array_metadata -from zarr.core.buffer.core import default_buffer_prototype -from zarr.errors import NodeNotFoundError -from zarr.zarrs import ( - NodeExistsError, - create_new_array, - create_new_group, - create_overwrite_array, - create_overwrite_group, - delete_node, - list_children, - read_metadata, -) - -if TYPE_CHECKING: - from zarr.abc.store import Store - -GROUP_META: dict[str, Any] = { - "zarr_format": 3, - "node_type": "group", - "attributes": {"answer": 42}, -} - - -async def test_create_new_group(store: Store) -> None: - await create_new_group(GROUP_META, store, "foo") - group = zarr.open_group(store=store, path="foo", mode="r") - assert dict(group.attrs) == {"answer": 42} - - -async def test_create_new_group_at_root(store: Store) -> None: - await create_new_group(GROUP_META, store, "") - group = zarr.open_group(store=store, mode="r") - assert dict(group.attrs) == {"answer": 42} - - -async def test_create_new_group_existing_node(store: Store) -> None: - await create_new_group(GROUP_META, store, "foo") - with pytest.raises(NodeExistsError): - await create_new_group(GROUP_META, store, "foo") - - -async def test_create_overwrite_group(store: Store) -> None: - # an array and its chunks previously occupied the path; overwrite removes both - arr = zarr.create_array(store=store, name="foo", shape=(4,), chunks=(2,), dtype="uint8") - arr[:] = 1 - assert await store.exists("foo/c/0") - await create_overwrite_group(GROUP_META, store, "foo") - group = zarr.open_group(store=store, path="foo", mode="r") - assert dict(group.attrs) == {"answer": 42} - assert not await store.exists("foo/c/0") - assert await store.get("foo/zarr.json", prototype=default_buffer_prototype()) is not None - - -async def test_create_new_array(store: Store) -> None: - await create_new_array(array_metadata(), store, "arr") - arr = zarr.open_array(store=store, path="arr", mode="r") - assert arr.shape == (8, 8) - assert arr.chunks == (4, 4) - assert arr.dtype == np.dtype("uint16") - - -async def test_create_new_array_existing_node(store: Store) -> None: - await create_new_array(array_metadata(), store, "arr") - with pytest.raises(NodeExistsError): - await create_new_array(array_metadata(), store, "arr") - - -async def test_create_overwrite_array(store: Store) -> None: - zarr.create_group(store=store, path="arr") - await create_overwrite_array(array_metadata(), store, "arr") - arr = zarr.open_array(store=store, path="arr", mode="r") - assert arr.shape == (8, 8) - - -async def test_create_new_array_v2(store: Store) -> None: - await create_new_array(array_metadata(zarr_format=2), store, "arr") - arr = zarr.open_array(store=store, path="arr", mode="r") - assert arr.metadata.zarr_format == 2 - assert arr.shape == (8, 8) - - -async def test_read_metadata_matches_stored_document(store: Store) -> None: - await create_new_array(array_metadata(), store, "arr") - observed = await read_metadata(store, "arr") - raw = await store.get("arr/zarr.json", prototype=default_buffer_prototype()) - assert raw is not None - assert observed == json.loads(raw.to_bytes()) - - -async def test_read_metadata_zarr_python_group(store: Store) -> None: - zarr.create_group(store=store, path="g", attributes={"a": 1}) - observed = await read_metadata(store, "g") - assert observed["node_type"] == "group" - assert observed["attributes"] == {"a": 1} - - -async def test_read_metadata_missing(store: Store) -> None: - with pytest.raises(NodeNotFoundError): - await read_metadata(store, "nope") - - -async def test_delete_node(store: Store) -> None: - arr = zarr.create_array(store=store, name="doomed", shape=(4,), chunks=(2,), dtype="uint8") - arr[:] = 1 - await delete_node(store, "doomed") - assert not await store.exists("doomed/zarr.json") - assert not await store.exists("doomed/c/0") - - -async def test_delete_node_missing(store: Store) -> None: - with pytest.raises(NodeNotFoundError): - await delete_node(store, "nope") - - -async def test_list_children(store: Store) -> None: - root = zarr.create_group(store=store) - root.create_group("sub_group", attributes={"kind": "group"}) - root.create_array("sub_array", shape=(4,), chunks=(2,), dtype="uint8") - children = await list_children(store, "") - by_path = dict(children) - assert set(by_path) == {"sub_group", "sub_array"} - assert not any(p.startswith("/") for p in by_path) - assert by_path["sub_group"]["node_type"] == "group" - assert by_path["sub_array"]["node_type"] == "array" - - -async def test_list_children_missing(store: Store) -> None: - with pytest.raises(NodeNotFoundError): - await list_children(store, "nope") From 03c80d743c820071fab3d4e47210f5b5836c8201 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 18:44:21 +0200 Subject: [PATCH 38/75] docs/ci: zarr.crud changelog and CI coverage Co-Authored-By: Claude Fable 5 --- .github/workflows/zarrs.yml | 2 +- changes/+zarrs-bindings.feature.md | 15 +++++++++------ 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/.github/workflows/zarrs.yml b/.github/workflows/zarrs.yml index 954dcda79b..b17cfb3a3b 100644 --- a/.github/workflows/zarrs.yml +++ b/.github/workflows/zarrs.yml @@ -31,4 +31,4 @@ jobs: - name: Run zarrs bindings tests # the ubuntu runner image ships a Rust toolchain; the maturin build # backend is fetched by uv on demand - run: uv run --group zarrs pytest tests/zarrs -v + run: uv run --group zarrs pytest tests/crud tests/zarrs -v diff --git a/changes/+zarrs-bindings.feature.md b/changes/+zarrs-bindings.feature.md index 0b0e7ee384..26216a71ae 100644 --- a/changes/+zarrs-bindings.feature.md +++ b/changes/+zarrs-bindings.feature.md @@ -1,6 +1,9 @@ -Added `zarr.zarrs`, an experimental low-level functional API for zarr hierarchy -CRUD backed by the Rust [zarrs](https://zarrs.dev) crate via the new in-repo -`zarrs-bindings` PyO3 crate. Array routines take an explicit metadata document, -enabling read-only views such as decoding chunks with externally supplied -metadata or reading raw encoded chunk bytes. Build for development with -`uv sync --group zarrs`. +Added `zarr.crud`, an experimental backend-agnostic low-level functional API for +zarr hierarchy CRUD (`create_*`, `read_chunk`, `read_region`, `read_encoded_chunk`, +`write_chunk`, `delete_chunk`, `read_metadata`, `delete_node`, `list_children`). +Array routines take an explicit metadata document, enabling read-only views. +Operations delegate to a pluggable `CrudBackend`: a pure-Python reference backend +(the default) or the zarrs-accelerated backend in `zarr.zarrs`, backed by the Rust +[zarrs](https://zarrs.dev) crate via the in-repo `zarrs-bindings` PyO3 crate. +Select a backend with the `crud.backend` config key or a per-call `backend=` +argument. Build the zarrs backend for development with `uv sync --group zarrs`. From c8ce00e5e84bdf2b7efdd4d73dee8216b6d69ba6 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 19:19:38 +0200 Subject: [PATCH 39/75] fix: reference backend matches zarrs on v2 groups and all-fill chunk drops - Add ZGROUP_JSON + a _node_exists() helper probing all three meta-key variants (zarr.json / .zarray / .zgroup) so v2 groups are visible to every CRUD operation; rewrite _create/read_metadata/delete_node/list_children to use it; remove dead _meta_key(). - Add _is_all_fill_value() and drop all-fill chunks in write_chunk to match zarrs's sparse-storage convention. - Add differential tests for v2 groups, v2 array read_metadata, and all-fill chunk writes. Co-Authored-By: Claude Fable 5 --- src/zarr/crud/_reference.py | 84 ++++++++++++++++++------------------- tests/crud/test_crud.py | 38 +++++++++++++++++ 2 files changed, 78 insertions(+), 44 deletions(-) diff --git a/src/zarr/crud/_reference.py b/src/zarr/crud/_reference.py index 2b48186e56..78db4c6e04 100644 --- a/src/zarr/crud/_reference.py +++ b/src/zarr/crud/_reference.py @@ -7,7 +7,7 @@ from zarr.core.array import AsyncArray, create_codec_pipeline from zarr.core.array_spec import ArrayConfig, ArraySpec from zarr.core.buffer.core import NDBuffer, default_buffer_prototype -from zarr.core.common import ZARR_JSON, ZARRAY_JSON, ZATTRS_JSON +from zarr.core.common import ZARR_JSON, ZARRAY_JSON, ZATTRS_JSON, ZGROUP_JSON from zarr.core.group import GroupMetadata from zarr.core.metadata.io import save_metadata from zarr.core.metadata.v2 import ArrayV2Metadata @@ -49,10 +49,16 @@ def _array_spec(meta_obj: ArrayV3Metadata | ArrayV2Metadata, shape: tuple[int, . ) -def _meta_key(path: str, zarr_format: int) -> str: - fname = ZARR_JSON if zarr_format == 3 else ZARRAY_JSON - p = path.strip("/") - return f"{p}/{fname}" if p else fname +def _is_all_fill_value( + arr: np.ndarray[Any, np.dtype[Any]], fill_value: Any, dtype: np.dtype[Any] +) -> bool: + """Whether every element of `arr` equals the fill value (NaN-aware for floats).""" + if fill_value is None: + return False + fill = np.asarray(fill_value, dtype=dtype) + if np.issubdtype(dtype, np.floating) or np.issubdtype(dtype, np.complexfloating): + return bool(np.array_equal(arr, np.broadcast_to(fill, arr.shape), equal_nan=True)) + return bool(np.all(arr == fill)) class ReferenceBackend: @@ -63,47 +69,50 @@ class ReferenceBackend: subset reads, which is exactly the `BasicIndexer` + codec-pipeline read path. """ + async def _node_exists(self, store: Store, path: str) -> bool: + proto = default_buffer_prototype() + sp = StorePath(store, path.strip("/")) + for meta_key in (ZARR_JSON, ZARRAY_JSON, ZGROUP_JSON): + if await (sp / meta_key).get(prototype=proto) is not None: + return True + return False + async def create_array( self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool ) -> None: meta_obj = parse_array_metadata(metadata) - await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) + await self._create(store, path, meta_obj, overwrite=overwrite) async def create_group( self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool ) -> None: meta_obj = GroupMetadata.from_dict(dict(metadata)) - await self._create(store, path, meta_obj, meta_obj.zarr_format, overwrite=overwrite) + await self._create(store, path, meta_obj, overwrite=overwrite) - async def _create( - self, store: Store, path: str, meta_obj: Any, zarr_format: int, *, overwrite: bool - ) -> None: + async def _create(self, store: Store, path: str, meta_obj: Any, *, overwrite: bool) -> None: sp = StorePath(store, path.strip("/")) - proto = default_buffer_prototype() if overwrite: await store.delete_dir(path.strip("/")) - else: - key = _meta_key(path, zarr_format) - if await store.get(key, prototype=proto) is not None: - raise NodeExistsError(f"a node already exists at path {path!r}") + elif await self._node_exists(store, path): + raise NodeExistsError(f"a node already exists at path {path!r}") await save_metadata(sp, meta_obj, ensure_parents=True) async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: from zarr.core._json import buffer_to_json_object proto = default_buffer_prototype() - p = path.strip("/") - sp = StorePath(store, p) + sp = StorePath(store, path.strip("/")) buf = await (sp / ZARR_JSON).get(prototype=proto) if buf is not None: return buffer_to_json_object(buf) - buf2 = await (sp / ZARRAY_JSON).get(prototype=proto) - if buf2 is not None: - doc = buffer_to_json_object(buf2) - zattrs = await (sp / ZATTRS_JSON).get(prototype=proto) - if zattrs is not None: - doc["attributes"] = buffer_to_json_object(zattrs) - return doc + for meta_key in (ZARRAY_JSON, ZGROUP_JSON): + b = await (sp / meta_key).get(prototype=proto) + if b is not None: + doc = buffer_to_json_object(b) + zattrs = await (sp / ZATTRS_JSON).get(prototype=proto) + if zattrs is not None: + doc["attributes"] = buffer_to_json_object(zattrs) + return doc raise NodeNotFoundError(f"no node found at path {path!r}") async def read_chunk( @@ -157,6 +166,9 @@ async def write_chunk( sp = StorePath(store, path.strip("/")) chunk_key = meta_obj.encode_chunk_key(coords) arr = np.frombuffer(data, dtype=np_dtype).reshape(shape) + if _is_all_fill_value(arr, meta_obj.fill_value, np_dtype): + await (sp / chunk_key).delete() + return pipeline = create_codec_pipeline(meta_obj) spec = _array_spec(meta_obj, shape) encoded = list(await pipeline.encode([(NDBuffer.from_ndarray_like(arr), spec)])) @@ -174,34 +186,18 @@ async def delete_chunk( await (sp / meta_obj.encode_chunk_key(coords)).delete() async def delete_node(self, store: Store, path: str) -> None: - proto = default_buffer_prototype() - p = path.strip("/") - sp = StorePath(store, p) - present = ( - await (sp / ZARR_JSON).get(prototype=proto) is not None - or await (sp / ZARRAY_JSON).get(prototype=proto) is not None - ) - if not present: + if not await self._node_exists(store, path): raise NodeNotFoundError(f"no node found at path {path!r}") - await store.delete_dir(p) + await store.delete_dir(path.strip("/")) async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: - proto = default_buffer_prototype() p = path.strip("/") - sp = StorePath(store, p) - if ( - await (sp / ZARR_JSON).get(prototype=proto) is None - and await (sp / ZARRAY_JSON).get(prototype=proto) is None - ): + if not await self._node_exists(store, path): raise NodeNotFoundError(f"no node found at path {path!r}") prefix = f"{p}/" if p else "" children: list[tuple[str, dict[str, JSON]]] = [] async for name in store.list_dir(prefix): child_path = f"{p}/{name}" if p else name - child_sp = StorePath(store, child_path) - if ( - await (child_sp / ZARR_JSON).get(prototype=proto) is not None - or await (child_sp / ZARRAY_JSON).get(prototype=proto) is not None - ): + if await self._node_exists(store, child_path): children.append((name, await self.read_metadata(store, child_path))) return children diff --git a/tests/crud/test_crud.py b/tests/crud/test_crud.py index 2b406c0a5f..d58256c1d7 100644 --- a/tests/crud/test_crud.py +++ b/tests/crud/test_crud.py @@ -32,6 +32,7 @@ from zarr.abc.store import Store GROUP_META: dict[str, Any] = {"zarr_format": 3, "node_type": "group", "attributes": {"answer": 42}} +GROUP_META_V2: dict[str, Any] = {"zarr_format": 2, "attributes": {"answer": 42}} # --- node lifecycle --- @@ -111,6 +112,23 @@ async def test_list_children(backend: str, store: Store) -> None: assert not any(p.startswith("/") for p in by_path) +async def test_create_read_delete_v2_group(backend: str, store: Store) -> None: + await create_new_group(GROUP_META_V2, store, "g2", backend=backend) + meta = await read_metadata(store, "g2", backend=backend) + assert meta["zarr_format"] == 2 + with pytest.raises(NodeExistsError): + await create_new_group(GROUP_META_V2, store, "g2", backend=backend) + await delete_node(store, "g2", backend=backend) + with pytest.raises(NodeNotFoundError): + await read_metadata(store, "g2", backend=backend) + + +async def test_read_metadata_v2_array(backend: str, store: Store) -> None: + await create_new_array(array_metadata(zarr_format=2), store, "arr", backend=backend) + meta = await read_metadata(store, "arr", backend=backend) + assert meta["zarr_format"] == 2 + + # --- chunk I/O --- @@ -197,6 +215,26 @@ async def test_delete_chunk(backend: str, store: Store) -> None: assert not await store.exists("a/c/0/0") +async def test_write_all_fill_chunk_is_dropped(backend: str, store: Store) -> None: + arr = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=0 + ) + meta = dict(arr.metadata.to_dict()) + await write_chunk(meta, store, "a", (0, 0), np.zeros((4, 4), dtype="uint16"), backend=backend) + assert not await store.exists("a/c/0/0") + np.testing.assert_array_equal( + await read_chunk(meta, store, "a", (0, 0), backend=backend), + np.zeros((4, 4), dtype="uint16"), + ) + + +async def test_overwrite_chunk_with_fill_removes_it(backend: str, store: Store) -> None: + _data, meta = filled(store) # chunk (0,0) exists with nonzero data, fill_value default 0 + assert await store.exists("a/c/0/0") + await write_chunk(meta, store, "a", (0, 0), np.zeros((4, 4), dtype="uint16"), backend=backend) + assert not await store.exists("a/c/0/0") + + async def test_read_encoded_chunk_matches_store(backend: str, store: Store) -> None: _, meta = filled(store) raw = await read_encoded_chunk(meta, store, "a", (0, 0), backend=backend) From 75d44aec5e0b35580b2a8a802640157d08423294 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Mon, 15 Jun 2026 19:27:04 +0200 Subject: [PATCH 40/75] test: track zarrs v2-group attribute divergence as strict xfail; document it Co-Authored-By: Claude Fable 5 --- src/zarr/zarrs/_backend.py | 7 +++++++ tests/crud/test_crud.py | 21 +++++++++++++++++++++ 2 files changed, 28 insertions(+) diff --git a/src/zarr/zarrs/_backend.py b/src/zarr/zarrs/_backend.py index 196cc10af0..e95759660b 100644 --- a/src/zarr/zarrs/_backend.py +++ b/src/zarr/zarrs/_backend.py @@ -41,6 +41,13 @@ class ZarrsBackend: the `/`-prefixed node-path form, store resolution, offloading the blocking Rust calls to a worker thread, and translating binding exceptions to the canonical `zarr.crud` / `zarr.errors` types. + + Known limitation: creating a Zarr v2 *group* with attributes writes a + non-standard `.zattrs` (the attributes nested under an ``"attributes"`` key) + that zarr-python and other readers interpret incorrectly. This is a + zarrs-crate behavior; the pure-Python reference backend writes the standard + layout. Prefer the reference backend for writing v2 groups until the zarrs + crate is fixed. """ async def create_array( diff --git a/tests/crud/test_crud.py b/tests/crud/test_crud.py index d58256c1d7..d4aa79e334 100644 --- a/tests/crud/test_crud.py +++ b/tests/crud/test_crud.py @@ -43,6 +43,27 @@ async def test_create_new_group(backend: str, store: Store) -> None: assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} +async def test_v2_group_attrs_zarr_python_compatible_reference(store: Store) -> None: + # The reference backend writes standard v2 `.zattrs` (the bare attributes + # dict), so zarr-python and other readers see the right attributes. + await create_new_group(GROUP_META_V2, store, "g2", backend="reference") + assert dict(zarr.open_group(store=store, path="g2", mode="r").attrs) == {"answer": 42} + + +@pytest.mark.xfail( + reason="the zarrs backend writes v2 group attributes in a non-standard `.zattrs` " + "layout (nested under an 'attributes' key) that zarr-python reads back wrong; " + "tracked zarrs-crate limitation", + strict=True, +) +async def test_v2_group_attrs_zarr_python_compatible_zarrs(store: Store) -> None: + pytest.importorskip("_zarrs_bindings", reason="zarrs-bindings is not installed") + import zarr.zarrs + + await create_new_group(GROUP_META_V2, store, "g2", backend="zarrs") + assert dict(zarr.open_group(store=store, path="g2", mode="r").attrs) == {"answer": 42} + + async def test_create_new_group_existing_raises(backend: str, store: Store) -> None: await create_new_group(GROUP_META, store, "foo", backend=backend) with pytest.raises(NodeExistsError): From 0094048efca9e71af8a6fae1a4d0f95c962eb7d8 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Tue, 16 Jun 2026 09:07:58 +0200 Subject: [PATCH 41/75] refactor: move zarrs-bindings crate under packages/ (repo convention) Matches the existing packages/zarr-metadata subpackage layout. Updates the uv source path, sdist exclude, gitignore, and design doc accordingly. Co-Authored-By: Claude Fable 5 --- .gitignore | 2 +- .../specs/2026-06-11-zarrs-functional-api-design.md | 3 ++- {zarrs-bindings => packages/zarrs-bindings}/Cargo.lock | 0 {zarrs-bindings => packages/zarrs-bindings}/Cargo.toml | 0 {zarrs-bindings => packages/zarrs-bindings}/pyproject.toml | 0 {zarrs-bindings => packages/zarrs-bindings}/src/chunk.rs | 0 {zarrs-bindings => packages/zarrs-bindings}/src/lib.rs | 0 {zarrs-bindings => packages/zarrs-bindings}/src/node.rs | 0 {zarrs-bindings => packages/zarrs-bindings}/src/store.rs | 0 pyproject.toml | 4 ++-- uv.lock | 4 ++-- 11 files changed, 7 insertions(+), 6 deletions(-) rename {zarrs-bindings => packages/zarrs-bindings}/Cargo.lock (100%) rename {zarrs-bindings => packages/zarrs-bindings}/Cargo.toml (100%) rename {zarrs-bindings => packages/zarrs-bindings}/pyproject.toml (100%) rename {zarrs-bindings => packages/zarrs-bindings}/src/chunk.rs (100%) rename {zarrs-bindings => packages/zarrs-bindings}/src/lib.rs (100%) rename {zarrs-bindings => packages/zarrs-bindings}/src/node.rs (100%) rename {zarrs-bindings => packages/zarrs-bindings}/src/store.rs (100%) diff --git a/.gitignore b/.gitignore index e5474b7c1c..ae184fa731 100644 --- a/.gitignore +++ b/.gitignore @@ -96,4 +96,4 @@ zarr.egg-info/ packages/zarr-metadata/uv.lock # zarrs-bindings Rust build artifacts -zarrs-bindings/target/ +packages/zarrs-bindings/target/ diff --git a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md index 0c7deb8eac..22cb0b5785 100644 --- a/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md +++ b/docs/superpowers/specs/2026-06-11-zarrs-functional-api-design.md @@ -39,7 +39,8 @@ codec-pipeline registry through this API (possible later), fancy Two distributions in this repo, hard boundary between them: -1. **Rust crate `zarrs-bindings`** at the repo root (`zarrs-bindings/`), +1. **Rust crate `zarrs-bindings`** under `packages/` (`packages/zarrs-bindings/`, + alongside the existing `zarr-metadata` subpackage), built with maturin (PyO3, `abi3-py312`), publishing wheel `zarrs-bindings` with native module `_zarrs_bindings`. It is a thin, mechanical binding over `zarrs`: functions/pyclasses take metadata as a **JSON string**, a diff --git a/zarrs-bindings/Cargo.lock b/packages/zarrs-bindings/Cargo.lock similarity index 100% rename from zarrs-bindings/Cargo.lock rename to packages/zarrs-bindings/Cargo.lock diff --git a/zarrs-bindings/Cargo.toml b/packages/zarrs-bindings/Cargo.toml similarity index 100% rename from zarrs-bindings/Cargo.toml rename to packages/zarrs-bindings/Cargo.toml diff --git a/zarrs-bindings/pyproject.toml b/packages/zarrs-bindings/pyproject.toml similarity index 100% rename from zarrs-bindings/pyproject.toml rename to packages/zarrs-bindings/pyproject.toml diff --git a/zarrs-bindings/src/chunk.rs b/packages/zarrs-bindings/src/chunk.rs similarity index 100% rename from zarrs-bindings/src/chunk.rs rename to packages/zarrs-bindings/src/chunk.rs diff --git a/zarrs-bindings/src/lib.rs b/packages/zarrs-bindings/src/lib.rs similarity index 100% rename from zarrs-bindings/src/lib.rs rename to packages/zarrs-bindings/src/lib.rs diff --git a/zarrs-bindings/src/node.rs b/packages/zarrs-bindings/src/node.rs similarity index 100% rename from zarrs-bindings/src/node.rs rename to packages/zarrs-bindings/src/node.rs diff --git a/zarrs-bindings/src/store.rs b/packages/zarrs-bindings/src/store.rs similarity index 100% rename from zarrs-bindings/src/store.rs rename to packages/zarrs-bindings/src/store.rs diff --git a/pyproject.toml b/pyproject.toml index d24917bc5f..89e05068ea 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,7 @@ exclude = [ "/.github", "/bench", "/docs", - "/zarrs-bindings", + "/packages/zarrs-bindings", ] [project] @@ -499,4 +499,4 @@ ignore-words-list = "astroid" zarr = "zarr.testing" [tool.uv.sources] -zarrs-bindings = { path = "zarrs-bindings" } +zarrs-bindings = { path = "packages/zarrs-bindings" } diff --git a/uv.lock b/uv.lock index a1c94a2d4b..838feef237 100644 --- a/uv.lock +++ b/uv.lock @@ -4190,9 +4190,9 @@ zarrs = [ { name = "pytest-xdist" }, { name = "tomlkit" }, { name = "uv" }, - { name = "zarrs-bindings", directory = "zarrs-bindings" }, + { name = "zarrs-bindings", directory = "packages/zarrs-bindings" }, ] [[package]] name = "zarrs-bindings" -source = { directory = "zarrs-bindings" } +source = { directory = "packages/zarrs-bindings" } From 220c5c73daf84f45549ee566f7b9993eccdd7c78 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Tue, 16 Jun 2026 09:51:53 +0200 Subject: [PATCH 42/75] docs: name changelog fragment with PR number (#4064) for the changelog check Co-Authored-By: Claude Fable 5 --- changes/{+zarrs-bindings.feature.md => 4064.feature.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changes/{+zarrs-bindings.feature.md => 4064.feature.md} (100%) diff --git a/changes/+zarrs-bindings.feature.md b/changes/4064.feature.md similarity index 100% rename from changes/+zarrs-bindings.feature.md rename to changes/4064.feature.md From 5ba9663704bce2872a3333b72572cd9d8cdbc062 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 12:18:06 +0200 Subject: [PATCH 43/75] docs: design spec for the array engine protocol (zarrista-backed) Assisted-by: ClaudeCode:claude-fable-5 --- ...2026-07-22-array-engine-protocol-design.md | 255 ++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md diff --git a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md new file mode 100644 index 0000000000..a47221916e --- /dev/null +++ b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md @@ -0,0 +1,255 @@ +# Array engine protocol for zarr-python + +Date: 2026-07-22 +Status: approved +Branch: `claude/zarr-array-engine-protocol-239cfa` + +## Goal + +Move the pluggable-backend boundary inside zarr-python's array classes: +`Array` wraps an object satisfying `ArrayEngineProtocol`, and `AsyncArray` +wraps an object satisfying `AsyncArrayEngineProtocol` (the same contract with +async methods). The engine owns the I/O data path — reading and writing +decoded data for selections — while `Array`/`AsyncArray` keep metadata +management, indexing normalization, and resize/append logic. + +Two engine families ship: + +- **Default engines** — the existing codec-pipeline machinery, moved behind + the protocol. Always available, every store, Zarr v2 and v3. Behavior and + performance are unchanged. +- **Zarrista engines** — powered by [Zarrista](https://pypi.org/project/zarrista/) + (Development Seed's zarrs-backed low-level Zarr API). We build against + Zarrista's `main` branch (unreleased; approved for use), pinned to a git + commit until the next release. + +This strategy **replaces** the `zarr.crud` layer and the homemade Rust +bindings from earlier on this branch. `zarr.crud`, `zarr.zarrs`, and the +`packages/zarrs-bindings` crate are deleted (see Deletions). + +Non-goals for this change: + +- Group-level engine operations (listing, group creation). The hierarchy + engine exists only to mint array engines in v1. +- Engine-owned metadata mutation (resize/append/update_attributes stay in + `Array`/`AsyncArray`). +- Silent fallback between engines (see Error handling). + +## Background: what Zarrista provides (main branch) + +Sync `Array` and async `AsyncArray` are separate native classes: + +- Construction: `open(store, path)` and `from_metadata(metadata, store, path)` + (constructs without writing; `store_metadata()` persists). Metadata is typed + as `zarr_metadata.ArrayMetadataV3` — our in-repo `packages/zarr-metadata` + package, so `ArrayV3Metadata.to_dict()` output is already the right currency. +- Reads: `retrieve_array_subset(selection)` (numpy basic indexing, step-1 + slices, ndim-preserving), `retrieve_chunk`, `retrieve_encoded_chunk`, and + sharding-aware `retrieve_subchunk` variants. +- Writes: `store_chunk` (decoded input as `ArrayBytes`; drops fill-value + chunks), `store_encoded_chunk`, `erase_chunk`, `compact_chunk`. **No + multi-chunk write** (`store_array_subset` is not exposed yet). +- Results: `DecodedArray = Tensor | VariableArray | MaskedTensor | + MaskedVariableArray`. `Tensor` is zero-copy to numpy (buffer protocol, + `__array__`, DLPack); `VariableArray` exports Arrow capsules. +- Stores: sync side takes Rust-native `FilesystemStore` / `MemoryStore`; + async side takes an obstore `ObjectStore` or an icechunk `Session`. There is + no bridge for arbitrary Python store objects. +- Zarr v3 only. + +## Architecture + +### Protocols (`zarr.abc.engine`) + +Four runtime-checkable protocols. The async pair: + +```python +class AsyncArrayEngineProtocol(Protocol): + async def read_selection( + self, + indexer: Indexer, + *, + prototype: BufferPrototype, + out: NDBuffer | None = None, + ) -> NDBuffer: ... + + async def write_selection( + self, + indexer: Indexer, + value: NDBuffer, + *, + prototype: BufferPrototype, + ) -> None: ... + + def with_metadata(self, metadata: ArrayMetadata) -> AsyncArrayEngineProtocol: ... + + +class AsyncHierarchyEngineProtocol(Protocol): + def array_engine( + self, path: str, metadata: ArrayMetadata + ) -> AsyncArrayEngineProtocol: ... +``` + +`ArrayEngineProtocol` and `HierarchyEngineProtocol` are identical with sync +methods and sync return types. + +Contract details: + +- An **array engine is bound** to `(store, path, metadata)` at construction. + Methods take only selection-level arguments, so engines can hold expensive + per-array state (a constructed `zarrista.Array`, codec pipelines, caches). +- A **hierarchy engine is bound to a store** and mints array engines that + share resources (the translated store handle, runtime, caches). This is the + factory the registry uses; users may also construct an array engine directly + and pass it to array creation/open. +- `read_selection` fills and returns `out` when given; when `out is None` it + returns a buffer of its own making (zero-copy where the backend allows). + The **engine speaks `Indexer`**: zarr-python's normalized selection plan + (iterable of `(chunk_coords, chunk_selection, out_selection, + is_complete_chunk)` projections with `shape` / `drop_axes`). Engines may + consume the indexer wholesale (fast paths) or iterate its projections. +- `with_metadata` returns a rebound engine for the same store/path with new + metadata. `resize`/`append`/`update_attributes` use it, so rebinding works + uniformly for factory-made and user-provided engines. +- Facade-side responsibilities (not the engine's): building the indexer, + `fields` handling, output dtype/order resolution, scalar extraction, + `drop_axes` squeezing, and the empty-selection short circuit. + +### Wiring into `Array`/`AsyncArray` + +- `AsyncArray` gains an `engine: AsyncArrayEngineProtocol` attribute. The + module-level `_get_selection`/`_set_selection` shrink to: resolve output + buffer arguments, call the engine, post-process. The current codec-pipeline + body of those functions **becomes** the default async engine's + implementation, byte-for-byte. +- `Array` gains `engine: ArrayEngineProtocol`. Its data methods + (`__getitem__`/`__setitem__`, `get/set_basic_selection`, + `get/set_orthogonal_selection`, `get/set_mask_selection`, + `get/set_coordinate_selection`, `get/set_block_selection`, and the + `oindex`/`vindex`/`blocks` accessors) call the sync engine directly — **the + sync data path no longer touches the event loop**. Non-I/O operations + (metadata, resize, create) keep the existing sync-over-async route. +- `DefaultAsyncArrayEngine` wraps store_path + metadata + codec pipeline + + config. `DefaultArrayEngine` adapts it with `sync()` — so the default sync + path is exactly as fast as today, while engines with native sync + implementations (Zarrista on local files) skip asyncio entirely. + +### Engine selection + +- `engine=` parameter on array creation/open APIs: an + `ArrayEngineProtocol`/`AsyncArrayEngineProtocol` instance (used as-is, sync + vs async matching the entry point), or a registered name (`"default"`, + `"zarrista"`), or `None`. +- `None` resolves via a hierarchy-engine registry keyed by name plus a config + key `array.engine` (default `"default"`). The registry maps a name to a + hierarchy-engine factory taking the zarr store; the resulting hierarchy + engine is cached per `(name, store)` so array engines opened from one store + share resources. +- This replaces the `zarr.crud` registry; the pattern (register by name, + config default, instance override) carries over. + +### Zarrista engines (`zarr.zarrista`) + +- `ZarristaHierarchyEngine` (sync) / `ZarristaAsyncHierarchyEngine`: + constructed with a zarr-python store, translating it at construction time: + - sync: `LocalStore → zarrista.FilesystemStore`. + - async: `zarr.storage.ObjectStore →` its underlying obstore instance; + an icechunk store/session → `icechunk.Session`. + - anything else (including zarr's `MemoryStore`, whose contents live in the + Python process): raise immediately with a message naming the store type + and the supported set. +- Array engines are built with + `zarrista.Array.from_metadata(metadata.to_dict(), store, path)` (async: + `AsyncArray.from_metadata`). Zarr v2 metadata raises immediately (Zarrista + is v3-only). +- **Reads**: an indexer that is basic with all-step-1 slices maps directly to + a zarrista `Selection` and one `retrieve_array_subset` call (all-Rust, + sharding-aware, parallel). Any other indexer (steps, orthogonal, + coordinate, mask, block) is served per-chunk: for each projection, + `retrieve_chunk` → zero-copy numpy view → scatter into the output buffer. + Rust still does all decoding; Python does the gather. +- **Writes**: per-chunk over the indexer's projections. A complete-chunk + projection encodes the value directly via `store_chunk`; a partial chunk + does read-modify-write: `retrieve_chunk` → patch with numpy → `store_chunk`. + When Zarrista exposes `store_array_subset`, the basic-selection fast path + upgrades without protocol changes. +- **Buffers**: `Tensor` → zero-copy numpy (`to_numpy`); `VariableArray` → + Arrow → numpy object array for vlen dtypes; `MaskedTensor`/ + `MaskedVariableArray` unsupported (zarr-python has no masked dtype) — raise. +- Registered under the name `"zarrista"` on import of `zarr.zarrista`; import + errors cleanly when `zarrista` is not installed. +- Dependency: optional dependency group `zarrista`, git-pinned to a Zarrista + `main` commit until its next release, then `>=` that release. + +## Error handling + +Fail loud, no silent fallback: + +- Engine/hierarchy-engine construction raises `UnsupportedEngineError` (new, + in `zarr.errors`) when the engine cannot serve the store or metadata + (untranslatable store, v2 metadata, a dtype that decodes to a masked + layout). The message names the incompatibility and the supported set. +- Operations an engine cannot perform raise `NotImplementedError`. +- Engines raise zarr-python's canonical exceptions where they exist + (`zarr.errors` types); Zarrista exceptions are translated at the engine + boundary. + +## Testing + +- `tests/engine/` differential suite: the same operations executed through + the default and zarrista engines with results compared — reads/writes over + basic (with and without steps), orthogonal, coordinate, mask, and block + selections; partial-chunk RMW; fill-value handling; sharded arrays; + vlen dtypes. Sync engines on `LocalStore`; async engines on an + obstore local store. Zarrista params skip when `zarrista` is not installed + (xdist-safe importorskip). +- Per the testing convention: one test function per operation family covering + reasonable input combinations, plus one test function per error case + (untranslatable store, v2 metadata, masked dtype, unsupported operation). +- Protocol conformance is checked statically: mypy verifies both default and + zarrista engines satisfy the protocols (`runtime_checkable` isinstance + checks only verify method presence). +- The sync zarrista path gets a no-event-loop regression test (engine methods + run with no running loop and never create one). +- CI: drop the Rust-crate build job; add a job installing the pinned Zarrista + and running `tests/engine`. + +## Deletions + +- `src/zarr/crud/` (protocol, facade, reference backend, registry). +- `src/zarr/zarrs/` and `packages/zarrs-bindings/` (the homemade PyO3 crate, + its construction cache, store shim, and bridge). +- `tests/crud/`, `tests/zarrs/`, and the crate-build CI wiring. +- Changelog fragments describing `zarr.crud`/`zarr.zarrs` are replaced by one + describing the engine protocol and the zarrista engine. + +The differential-testing idea from `zarr.crud` survives as `tests/engine/`; +the store-translation and metadata-handoff learnings carry into +`zarr.zarrista`. + +## Feature requests for Zarrista (tracked, not blocking) + +1. **`store_array_subset`** — zarrs has it natively; exposing it moves + partial-chunk read-modify-write into Rust and collapses the per-chunk + write path for basic selections. +2. **Python store bridge** — would lift the store-translation restriction and + let any zarr-python `Store` back a zarrista engine. +3. **Step > 1 slices in `Selection`** — widens the read fast path to strided + basic selections. +4. **Zarr v2 read support** — zarrs supports a v2 subset; exposing it would + let the zarrista engine serve v2 arrays. + +## Decision log + +- Engine protocols replace `zarr.crud`/`CrudBackend` entirely; homemade Rust + bindings dropped in favor of Zarrista (main branch approved). +- Engine scope is I/O only; metadata management stays in the array classes. +- Both acquisition paths: hierarchy engine as factory (resource sharing) and + user-provided array engines at creation time. +- Sync path is truly sync: `Array` calls its sync engine directly. +- Store bridging by translating known stores; fail loud otherwise. +- No fallback between engines; errors are immediate and specific. +- Region writes: protocol is selection-level; zarrista engine does + Python-side per-chunk RMW now, upgradeable when `store_array_subset` lands. +- Protocol granularity: engines speak `Indexer` (approach A), keeping the + default path unchanged and giving engines full selection information. From 2b88696685f646602607679f709a1c4bfeb84efa Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 12:44:55 +0200 Subject: [PATCH 44/75] docs: integrate zarrista-developer review feedback into engine spec Assisted-by: ClaudeCode:claude-fable-5 --- ...2026-07-22-array-engine-protocol-design.md | 112 ++++++++++++------ 1 file changed, 76 insertions(+), 36 deletions(-) diff --git a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md index a47221916e..98d99b2efa 100644 --- a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md +++ b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md @@ -3,13 +3,14 @@ Date: 2026-07-22 Status: approved Branch: `claude/zarr-array-engine-protocol-239cfa` +Reviewed-by: @kylebarron (Zarrista developer), 2026-07-22 — feedback integrated ## Goal Move the pluggable-backend boundary inside zarr-python's array classes: -`Array` wraps an object satisfying `ArrayEngineProtocol`, and `AsyncArray` -wraps an object satisfying `AsyncArrayEngineProtocol` (the same contract with -async methods). The engine owns the I/O data path — reading and writing +`Array` wraps an object satisfying the `ArrayEngine` protocol, and +`AsyncArray` wraps an object satisfying the `AsyncArrayEngine` protocol (the +same contract with async methods). The engine owns the I/O data path — reading and writing decoded data for selections — while `Array`/`AsyncArray` keep metadata management, indexing normalization, and resize/append logic. @@ -21,7 +22,8 @@ Two engine families ship: - **Zarrista engines** — powered by [Zarrista](https://pypi.org/project/zarrista/) (Development Seed's zarrs-backed low-level Zarr API). We build against Zarrista's `main` branch (unreleased; approved for use), pinned to a git - commit until the next release. + commit; the Zarrista team can publish a new beta release on request, at + which point the pin becomes `>=` that beta. This strategy **replaces** the `zarr.crud` layer and the homemade Rust bindings from earlier on this branch. `zarr.crud`, `zarr.zarrs`, and the @@ -61,17 +63,18 @@ Sync `Array` and async `AsyncArray` are separate native classes: ### Protocols (`zarr.abc.engine`) -Four runtime-checkable protocols. The async pair: +Four runtime-checkable protocols, named without a `Protocol` suffix +(`ArrayEngine(Protocol)` is enough; implementations get names like +`ZarristaEngine`). The async pair: ```python -class AsyncArrayEngineProtocol(Protocol): +class AsyncArrayEngine(Protocol): async def read_selection( self, indexer: Indexer, *, prototype: BufferPrototype, - out: NDBuffer | None = None, - ) -> NDBuffer: ... + ) -> NDArrayLike: ... async def write_selection( self, @@ -81,17 +84,17 @@ class AsyncArrayEngineProtocol(Protocol): prototype: BufferPrototype, ) -> None: ... - def with_metadata(self, metadata: ArrayMetadata) -> AsyncArrayEngineProtocol: ... + def with_metadata(self, metadata: ArrayMetadata) -> AsyncArrayEngine: ... -class AsyncHierarchyEngineProtocol(Protocol): +class AsyncHierarchyEngine(Protocol): def array_engine( self, path: str, metadata: ArrayMetadata - ) -> AsyncArrayEngineProtocol: ... + ) -> AsyncArrayEngine: ... ``` -`ArrayEngineProtocol` and `HierarchyEngineProtocol` are identical with sync -methods and sync return types. +`ArrayEngine` and `HierarchyEngine` are identical with sync methods and sync +return types. Contract details: @@ -102,27 +105,36 @@ Contract details: share resources (the translated store handle, runtime, caches). This is the factory the registry uses; users may also construct an array engine directly and pass it to array creation/open. -- `read_selection` fills and returns `out` when given; when `out is None` it - returns a buffer of its own making (zero-copy where the backend allows). - The **engine speaks `Indexer`**: zarr-python's normalized selection plan +- `read_selection` returns a buffer of the engine's own making: any object + implementing at least `__array__` (numpy coercion), zero-copy where the + backend allows. Additional export protocols (DLPack, Arrow) are welcome on + concrete engines but are not part of the contract. There is **no `out` + parameter** in v1; one may be added later as a performance improvement, and + the facade meanwhile serves user-supplied `out=` arguments by copying the + engine's result into them. +- The **engine speaks `Indexer`**: zarr-python's normalized selection plan (iterable of `(chunk_coords, chunk_selection, out_selection, is_complete_chunk)` projections with `shape` / `drop_axes`). Engines may consume the indexer wholesale (fast paths) or iterate its projections. + (Reviewer note: Kyle raised chunk-level indexing as a simpler v1 contract; + we are staying indexer-level per the approach decision — it is the only + shape that keeps the default path unchanged.) - `with_metadata` returns a rebound engine for the same store/path with new metadata. `resize`/`append`/`update_attributes` use it, so rebinding works uniformly for factory-made and user-provided engines. - Facade-side responsibilities (not the engine's): building the indexer, `fields` handling, output dtype/order resolution, scalar extraction, - `drop_axes` squeezing, and the empty-selection short circuit. + `drop_axes` squeezing, copying into a user-supplied `out=` buffer, and the + empty-selection short circuit. ### Wiring into `Array`/`AsyncArray` -- `AsyncArray` gains an `engine: AsyncArrayEngineProtocol` attribute. The +- `AsyncArray` gains an `engine: AsyncArrayEngine` attribute. The module-level `_get_selection`/`_set_selection` shrink to: resolve output buffer arguments, call the engine, post-process. The current codec-pipeline body of those functions **becomes** the default async engine's implementation, byte-for-byte. -- `Array` gains `engine: ArrayEngineProtocol`. Its data methods +- `Array` gains `engine: ArrayEngine`. Its data methods (`__getitem__`/`__setitem__`, `get/set_basic_selection`, `get/set_orthogonal_selection`, `get/set_mask_selection`, `get/set_coordinate_selection`, `get/set_block_selection`, and the @@ -136,22 +148,30 @@ Contract details: ### Engine selection -- `engine=` parameter on array creation/open APIs: an - `ArrayEngineProtocol`/`AsyncArrayEngineProtocol` instance (used as-is, sync - vs async matching the entry point), or a registered name (`"default"`, - `"zarrista"`), or `None`. -- `None` resolves via a hierarchy-engine registry keyed by name plus a config - key `array.engine` (default `"default"`). The registry maps a name to a - hierarchy-engine factory taking the zarr store; the resulting hierarchy - engine is cached per `(name, store)` so array engines opened from one store - share resources. -- This replaces the `zarr.crud` registry; the pattern (register by name, - config default, instance override) carries over. +- `engine=` parameter on array creation/open APIs, with three accepted forms: + the string `"default"`, the string `"zarrista"`, or an + `ArrayEngine`/`AsyncArrayEngine` instance the user constructed themselves + (used as-is, sync vs async matching the entry point): + + ```python + engine = ZarristaEngine(...) # user-constructed, full control + zarr.open_array(store, engine=engine) + zarr.open_array(store, engine="zarrista") # named, we construct it + ``` + +- **No config key in v1.** Omitting `engine=` means the default (pure-Python) + engine — today's behavior. A global config default can be added later. +- Name resolution is a small internal mapping from name to hierarchy-engine + factory taking the zarr store; the resulting hierarchy engine is cached per + `(name, store)` so array engines opened from one store share resources. + This replaces the `zarr.crud` registry. ### Zarrista engines (`zarr.zarrista`) -- `ZarristaHierarchyEngine` (sync) / `ZarristaAsyncHierarchyEngine`: - constructed with a zarr-python store, translating it at construction time: +- Array engines `ZarristaEngine` (sync) / `ZarristaAsyncEngine`, minted by + `ZarristaHierarchyEngine` / `ZarristaAsyncHierarchyEngine`: the hierarchy + engine is constructed with a zarr-python store, translating it at + construction time: - sync: `LocalStore → zarrista.FilesystemStore`. - async: `zarr.storage.ObjectStore →` its underlying obstore instance; an icechunk store/session → `icechunk.Session`. @@ -171,11 +191,13 @@ Contract details: - **Writes**: per-chunk over the indexer's projections. A complete-chunk projection encodes the value directly via `store_chunk`; a partial chunk does read-modify-write: `retrieve_chunk` → patch with numpy → `store_chunk`. - When Zarrista exposes `store_array_subset`, the basic-selection fast path - upgrades without protocol changes. + Kyle confirmed exposing `store_array_subset` (multi-chunk write) is easy on + the Zarrista side; when it lands, basic-selection writes collapse to one + Rust call with no protocol change. - **Buffers**: `Tensor` → zero-copy numpy (`to_numpy`); `VariableArray` → - Arrow → numpy object array for vlen dtypes; `MaskedTensor`/ - `MaskedVariableArray` unsupported (zarr-python has no masked dtype) — raise. + numpy via Zarrista's upcoming numpy export (a copy for now — confirmed + planned); `MaskedTensor`/`MaskedVariableArray` unsupported (zarr-python has + no masked dtype) — raise. - Registered under the name `"zarrista"` on import of `zarr.zarrista`; import errors cleanly when `zarrista` is not installed. - Dependency: optional dependency group `zarrista`, git-pinned to a Zarrista @@ -229,6 +251,9 @@ the store-translation and metadata-handoff learnings carry into ## Feature requests for Zarrista (tracked, not blocking) +Kyle has reviewed this list and is open to all of them; (1) is confirmed easy +and (a copying) numpy export for `VariableArray` is already planned. + 1. **`store_array_subset`** — zarrs has it natively; exposing it moves partial-chunk read-modify-write into Rust and collapses the per-chunk write path for basic selections. @@ -253,3 +278,18 @@ the store-translation and metadata-handoff learnings carry into Python-side per-chunk RMW now, upgradeable when `store_array_subset` lands. - Protocol granularity: engines speak `Indexer` (approach A), keeping the default path unchanged and giving engines full selection information. + +Post-review (kylebarron, on the shared gist): + +- Protocol names drop the `Protocol` suffix: `ArrayEngine`, + `AsyncArrayEngine`, `HierarchyEngine`, `AsyncHierarchyEngine`. +- No `out` parameter in the v1 protocol; revisit later for performance. +- Engine read results only need `__array__`; other export protocols + (DLPack, Arrow) are optional extras on implementations. +- No config key in v1: engine choice is explicit via + `engine="default" | "zarrista" | `; a global config default may + come later. +- Zarrista can cut a beta release on request, so the git pin is temporary. +- Kyle's open question — chunk-level indexing as a simpler v1 engine + contract — noted; staying indexer-level (approach A) to keep the default + path byte-for-byte unchanged. From efb3eb20096d03b3ca348027d5bdddfef55319c2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 13:01:35 +0200 Subject: [PATCH 45/75] docs: engine interchange is a contiguous Region box, not Indexer Assisted-by: ClaudeCode:claude-fable-5 --- ...2026-07-22-array-engine-protocol-design.md | 97 +++++++++++++------ 1 file changed, 66 insertions(+), 31 deletions(-) diff --git a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md index 98d99b2efa..fcb34d3629 100644 --- a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md +++ b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md @@ -68,17 +68,28 @@ Four runtime-checkable protocols, named without a `Protocol` suffix `ZarristaEngine`). The async pair: ```python +class Region(NamedTuple): + """A contiguous, step-1 box in array-element coordinates. + + One entry per dimension; `start` inclusive, `end_exclusive` exclusive, + already normalized (non-negative, clipped to the array shape). + """ + + start: tuple[int, ...] + end_exclusive: tuple[int, ...] + + class AsyncArrayEngine(Protocol): async def read_selection( self, - indexer: Indexer, + selection: Region, *, prototype: BufferPrototype, ) -> NDArrayLike: ... async def write_selection( self, - indexer: Indexer, + selection: Region, value: NDBuffer, *, prototype: BufferPrototype, @@ -112,28 +123,44 @@ Contract details: parameter** in v1; one may be added later as a performance improvement, and the facade meanwhile serves user-supplied `out=` arguments by copying the engine's result into them. -- The **engine speaks `Indexer`**: zarr-python's normalized selection plan - (iterable of `(chunk_coords, chunk_selection, out_selection, - is_complete_chunk)` projections with `shape` / `drop_axes`). Engines may - consume the indexer wholesale (fast paths) or iterate its projections. - (Reviewer note: Kyle raised chunk-level indexing as a simpler v1 contract; - we are staying indexer-level per the approach decision — it is the only - shape that keeps the default path unchanged.) +- The **engine speaks `Region`**: indexing across the engine boundary is + restricted to contiguous, step-1 boxes, interchanged as the `Region` + namedtuple above. Results and `value` inputs are ndim-preserving with shape + `end_exclusive - start` per dimension (matching Zarrista's ndim-preserving + `Selection` semantics). Engines never see raw user selections, steps, + fancy indices, or zarr-python `Indexer` objects. `write_selection` must + handle boxes that partially overlap chunks (read-modify-write is the + engine's concern; the default engine's codec pipeline already does this). - `with_metadata` returns a rebound engine for the same store/path with new metadata. `resize`/`append`/`update_attributes` use it, so rebinding works uniformly for factory-made and user-provided engines. -- Facade-side responsibilities (not the engine's): building the indexer, - `fields` handling, output dtype/order resolution, scalar extraction, - `drop_axes` squeezing, copying into a user-supplied `out=` buffer, and the - empty-selection short circuit. +- Facade-side responsibilities (not the engine's): normalizing every public + selection kind down to `Region` calls, `fields` handling, output + dtype/order resolution, scalar extraction and integer-axis squeezing, + copying into a user-supplied `out=` buffer, and the empty-selection short + circuit. Normalization rules: + - contiguous basic selections (slices with step 1, integers as length-1 + ranges, Ellipsis expansion) map to one `Region` directly; + - everything else — strided slices, orthogonal/coordinate/mask/block + selections — is served through its step-1 **bounding box**: reads fetch + the box and post-index the result; writes read the box, patch it with + numpy indexing, and write the box back (facade-level RMW). + - Consequence, accepted for now: sparse fancy selections over a large + extent transfer the whole bounding box through the engine. Revisit via a + richer interchange if it bites. ### Wiring into `Array`/`AsyncArray` - `AsyncArray` gains an `engine: AsyncArrayEngine` attribute. The - module-level `_get_selection`/`_set_selection` shrink to: resolve output - buffer arguments, call the engine, post-process. The current codec-pipeline - body of those functions **becomes** the default async engine's - implementation, byte-for-byte. + module-level `_get_selection`/`_set_selection` shrink to: normalize the + selection to `Region` calls (see normalization rules above), call the + engine, post-process. The default async engine implements + `read_selection`/`write_selection` by building a `BasicIndexer` for the + region and running today's codec-pipeline machinery — for contiguous + selections the work done is identical to today. For strided and fancy + selections the facade's bounding-box normalization replaces today's + per-chunk gather/scatter for **all** engines; this is a deliberate + simplification (see the accepted consequence above). - `Array` gains `engine: ArrayEngine`. Its data methods (`__getitem__`/`__setitem__`, `get/set_basic_selection`, `get/set_orthogonal_selection`, `get/set_mask_selection`, @@ -182,18 +209,16 @@ Contract details: `zarrista.Array.from_metadata(metadata.to_dict(), store, path)` (async: `AsyncArray.from_metadata`). Zarr v2 metadata raises immediately (Zarrista is v3-only). -- **Reads**: an indexer that is basic with all-step-1 slices maps directly to - a zarrista `Selection` and one `retrieve_array_subset` call (all-Rust, - sharding-aware, parallel). Any other indexer (steps, orthogonal, - coordinate, mask, block) is served per-chunk: for each projection, - `retrieve_chunk` → zero-copy numpy view → scatter into the output buffer. - Rust still does all decoding; Python does the gather. -- **Writes**: per-chunk over the indexer's projections. A complete-chunk - projection encodes the value directly via `store_chunk`; a partial chunk - does read-modify-write: `retrieve_chunk` → patch with numpy → `store_chunk`. - Kyle confirmed exposing `store_array_subset` (multi-chunk write) is easy on - the Zarrista side; when it lands, basic-selection writes collapse to one - Rust call with no protocol change. +- **Reads**: a `Region` maps directly to a zarrista `Selection` (a tuple of + step-1 slices) and one `retrieve_array_subset` call — all-Rust, + sharding-aware, parallel. No selection-kind dispatch in the engine at all. +- **Writes**: `write_selection` decomposes the region over the chunk grid + internally: a chunk fully covered by the region encodes the value directly + via `store_chunk`; a partially covered chunk does read-modify-write — + `retrieve_chunk` → patch with numpy → `store_chunk`. Kyle confirmed + exposing `store_array_subset` (multi-chunk write) is easy on the Zarrista + side; when it lands, `write_selection` collapses to one Rust call with no + protocol change. - **Buffers**: `Tensor` → zero-copy numpy (`to_numpy`); `VariableArray` → numpy via Zarrista's upcoming numpy export (a copy for now — confirmed planned); `MaskedTensor`/`MaskedVariableArray` unsupported (zarr-python has @@ -278,6 +303,8 @@ and (a copying) numpy export for `VariableArray` is already planned. Python-side per-chunk RMW now, upgradeable when `store_array_subset` lands. - Protocol granularity: engines speak `Indexer` (approach A), keeping the default path unchanged and giving engines full selection information. + **Superseded** — see the post-review revision below: the interchange is now + the contiguous-box `Region`. Post-review (kylebarron, on the shared gist): @@ -291,5 +318,13 @@ Post-review (kylebarron, on the shared gist): come later. - Zarrista can cut a beta release on request, so the git pin is temporary. - Kyle's open question — chunk-level indexing as a simpler v1 engine - contract — noted; staying indexer-level (approach A) to keep the default - path byte-for-byte unchanged. + contract — initially answered with indexer-level (approach A). + +Post-review revision (supersedes approach A's interchange): + +- The engine interchange is restricted to contiguous, step-1 boxes: the + `Region` namedtuple (`start`, `end_exclusive`). Engines never see raw + selections or `Indexer` objects. All selection normalization (and + bounding-box + post-index / facade-RMW for strided and fancy selections) + moves to the facade. The bounding-box amplification for sparse fancy + selections is an accepted trade-off for now. From e7d3947269e6c347f8c230ac0a4756ef1fcd1529 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 13:11:03 +0200 Subject: [PATCH 46/75] docs: implementation plan for the array engine protocol Assisted-by: ClaudeCode:claude-fable-5 --- .../plans/2026-07-22-array-engine-protocol.md | 2205 +++++++++++++++++ 1 file changed, 2205 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-22-array-engine-protocol.md diff --git a/docs/superpowers/plans/2026-07-22-array-engine-protocol.md b/docs/superpowers/plans/2026-07-22-array-engine-protocol.md new file mode 100644 index 0000000000..2eae8dd902 --- /dev/null +++ b/docs/superpowers/plans/2026-07-22-array-engine-protocol.md @@ -0,0 +1,2205 @@ +# Array Engine Protocol Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** `Array`/`AsyncArray` route all data I/O through pluggable engine objects (`ArrayEngine`/`AsyncArrayEngine` protocols, `Region` interchange), with a default pure-Python engine and a Zarrista (zarrs-backed) engine, replacing `zarr.crud`/`zarr.zarrs`/`packages/zarrs-bindings`. + +**Architecture:** Engines are bound to `(store, path, metadata)` and speak only contiguous step-1 boxes (`Region(start, end_exclusive)`). The facade (`Array`/`AsyncArray`) normalizes every public selection kind to Region calls: contiguous basic selections map directly; strided/orthogonal/coordinate/mask selections go through their bounding box with numpy post-indexing (reads) or box-level read-patch-write (writes). Hierarchy engines are store-bound factories minting resource-sharing array engines. The sync `Array` calls a sync engine directly (no event loop on the data path). + +**Tech Stack:** Python protocols (`typing.Protocol`), existing codec-pipeline machinery (default engine), Zarrista main branch (git-pinned; PyO3/zarrs) for the accelerated engine. + +**Spec:** `docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md` — read it before starting any task. + +## Global Constraints + +- Run all Python commands with `uv run` (e.g. `uv run pytest`, `uv run mypy`). +- Conventional commits with trailer `Assisted-by: ClaudeCode:`. +- Never `git add -A`; stage explicit paths only. Never commit `.claude/` or `.superpowers/`. +- Protocol names have no `Protocol` suffix: `ArrayEngine`, `AsyncArrayEngine`, `HierarchyEngine`, `AsyncHierarchyEngine`. +- The v1 engine contract has **no `out` parameter** and **no config key**; engine choice is explicit (`engine="default" | "zarrista" | `). +- Engine read results need only implement `__array__`. +- Fail loud: `UnsupportedEngineError` at engine construction; `NotImplementedError` for unsupported operations. No silent fallback. +- Docstrings are markdown (mkdocs), single backticks. +- Zarrista dependency: git-pinned to a `main` commit in an optional `zarrista` dependency group. + +## File Structure + +``` +src/zarr/abc/engine.py Region + the four protocols (new) +src/zarr/errors.py add UnsupportedEngineError +src/zarr/core/engine/__init__.py re-exports (new) +src/zarr/core/engine/_normalize.py selection-kind -> (Region, post_index) (new) +src/zarr/core/engine/_default.py DefaultArrayEngine / DefaultAsyncArrayEngine + hierarchy (new) +src/zarr/core/engine/_resolve.py engine-spec resolution + per-store hierarchy cache (new) +src/zarr/core/array.py facade rewiring (modify) +src/zarr/zarrista/__init__.py public zarrista engine surface (new) +src/zarr/zarrista/_translate.py zarr Store -> zarrista/obstore store translation (new) +src/zarr/zarrista/_engine.py ZarristaEngine / ZarristaAsyncEngine + hierarchy (new) +tests/engine/ protocol, normalization, default-engine, differential (new) +tests/zarrista/ translation + zarrista-only behavior (new) +DELETED: src/zarr/crud/, src/zarr/zarrs/, packages/zarrs-bindings/, tests/crud/, tests/zarrs/ +``` + +--- + +### Task 1: `Region`, the four protocols, `UnsupportedEngineError` + +**Files:** +- Create: `src/zarr/abc/engine.py` +- Modify: `src/zarr/errors.py` (append class) +- Test: `tests/engine/test_protocols.py`, `tests/engine/__init__.py` (empty) + +**Interfaces:** +- Produces: `Region(start: tuple[int, ...], end_exclusive: tuple[int, ...])` with property `shape -> tuple[int, ...]`; protocols `ArrayEngine`, `AsyncArrayEngine`, `HierarchyEngine`, `AsyncHierarchyEngine` exactly as below; `zarr.errors.UnsupportedEngineError(ValueError)`. Every later task imports these from `zarr.abc.engine` / `zarr.errors`. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/engine/test_protocols.py +from __future__ import annotations + +import numpy as np +import pytest + +from zarr.abc.engine import ArrayEngine, AsyncArrayEngine, Region +from zarr.errors import UnsupportedEngineError + + +def test_region() -> None: + """Region carries start/end_exclusive and derives shape.""" + r = Region(start=(1, 2), end_exclusive=(4, 2)) + assert r.start == (1, 2) + assert r.end_exclusive == (4, 2) + assert r.shape == (3, 0) + + +class _FakeSyncEngine: + def read_selection(self, selection, *, prototype): + return np.zeros(selection.shape) + + def write_selection(self, selection, value, *, prototype): + return None + + def with_metadata(self, metadata): + return self + + +def test_runtime_checkable_protocols() -> None: + """isinstance checks verify method presence for the sync protocol.""" + assert isinstance(_FakeSyncEngine(), ArrayEngine) + assert not isinstance(object(), ArrayEngine) + assert not isinstance(_FakeSyncEngine(), AsyncArrayEngine) or True # names match; mypy is authoritative + + +def test_unsupported_engine_error_is_value_error() -> None: + with pytest.raises(ValueError): + raise UnsupportedEngineError("nope") +``` + +- [ ] **Step 2: Run it to make sure it fails** + +Run: `uv run pytest tests/engine/test_protocols.py -v` +Expected: FAIL (`ModuleNotFoundError: zarr.abc.engine`) + +- [ ] **Step 3: Implement** + +```python +# src/zarr/abc/engine.py +"""Array engine protocols. + +An *array engine* owns the data path of one open array: reading and writing +decoded data for contiguous regions. `Array` wraps an object satisfying +`ArrayEngine`; `AsyncArray` wraps an object satisfying `AsyncArrayEngine`. +A *hierarchy engine* is bound to a store and mints array engines that share +resources. See `docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md`. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, NamedTuple, Protocol, runtime_checkable + +if TYPE_CHECKING: + from zarr.core.buffer import BufferPrototype, NDBuffer + from zarr.core.common import NDArrayLike + from zarr.core.metadata import ArrayMetadata + +__all__ = [ + "ArrayEngine", + "AsyncArrayEngine", + "AsyncHierarchyEngine", + "HierarchyEngine", + "Region", +] + + +class Region(NamedTuple): + """A contiguous, step-1 box in array-element coordinates. + + One entry per dimension; `start` is inclusive, `end_exclusive` exclusive. + Callers pass normalized values: non-negative and clipped to the array + shape. This is the only selection type that crosses the engine boundary. + """ + + start: tuple[int, ...] + end_exclusive: tuple[int, ...] + + @property + def shape(self) -> tuple[int, ...]: + """The ndim-preserving shape of the box.""" + return tuple(e - s for s, e in zip(self.start, self.end_exclusive, strict=True)) + + +@runtime_checkable +class ArrayEngine(Protocol): + """The synchronous data path of one open array. + + Bound to `(store, path, metadata)` at construction. Methods must not + require a running event loop. Read results are ndim-preserving with + shape `selection.shape` and need only implement `__array__`. + + Note: `runtime_checkable` isinstance checks only verify method names; + mypy is the authoritative conformance check. + """ + + def read_selection( + self, selection: Region, *, prototype: BufferPrototype + ) -> NDArrayLike: ... + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: ... + + def with_metadata(self, metadata: ArrayMetadata) -> ArrayEngine: ... + + +@runtime_checkable +class AsyncArrayEngine(Protocol): + """The asynchronous data path of one open array. See `ArrayEngine`.""" + + async def read_selection( + self, selection: Region, *, prototype: BufferPrototype + ) -> NDArrayLike: ... + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: ... + + def with_metadata(self, metadata: ArrayMetadata) -> AsyncArrayEngine: ... + + +@runtime_checkable +class HierarchyEngine(Protocol): + """A store-bound factory for synchronous array engines.""" + + def array_engine(self, path: str, metadata: ArrayMetadata) -> ArrayEngine: ... + + +@runtime_checkable +class AsyncHierarchyEngine(Protocol): + """A store-bound factory for asynchronous array engines.""" + + def array_engine(self, path: str, metadata: ArrayMetadata) -> AsyncArrayEngine: ... +``` + +Append to `src/zarr/errors.py` (match the module's existing docstring style): + +```python +class UnsupportedEngineError(ValueError): + """Raised when an array engine cannot serve the requested store or array. + + Examples: a store the engine cannot translate, or metadata (e.g. Zarr v2) + the engine does not support. Raised at engine construction time. + """ +``` + +Add `"UnsupportedEngineError"` to `zarr.errors.__all__` if the module defines one. + +- [ ] **Step 4: Run tests, then mypy** + +Run: `uv run pytest tests/engine/test_protocols.py -v` — Expected: PASS +Run: `uv run mypy src/zarr/abc/engine.py` — Expected: no errors + +- [ ] **Step 5: Commit** + +```bash +git add src/zarr/abc/engine.py src/zarr/errors.py tests/engine +git commit -m "feat: Region interchange and array/hierarchy engine protocols" +``` + +--- + +### Task 2: Selection normalization (`_normalize.py`) + +Every public selection kind reduces to `(Region, post_index)` where +`post_index` is a numpy index applied to the ndim-preserving box result. +The basic-kind code is adapted from `src/zarr/crud/_api.py::_normalize_selection` +(lines 86-147) — copy it before Task 9 deletes that package. + +**Files:** +- Create: `src/zarr/core/engine/__init__.py`, `src/zarr/core/engine/_normalize.py` +- Test: `tests/engine/test_normalize.py` + +**Interfaces:** +- Produces (all in `zarr.core.engine._normalize`; re-exported from `zarr.core.engine`): + - `normalize_basic(selection, shape) -> tuple[Region, tuple[slice | int, ...]]` + - `normalize_orthogonal(selection, shape) -> tuple[Region, tuple[Any, ...]]` (post is per-axis arrays/slices for `np.ix_`-style outer indexing; the helper returns the ready-to-use index) + - `normalize_coordinate(selection, shape) -> tuple[Region, tuple[np.ndarray, ...]]` + - `normalize_block(selection, chunk_grid_shape, chunk_shape, shape) -> Region` (block selections are contiguous; no post index) + - mask selections are converted by callers via `np.nonzero(mask)` → `normalize_coordinate`. +- Invariant used by Tasks 5-6: for any array `a`, + `np.asarray(engine_read(region))[post] == a[original_selection]`. + +- [ ] **Step 1: Write the failing tests** — differential against numpy, per your test convention (one combo test per function + error cases): + +```python +# tests/engine/test_normalize.py +from __future__ import annotations + +import numpy as np +import pytest + +from zarr.core.engine import ( + normalize_basic, + normalize_block, + normalize_coordinate, + normalize_orthogonal, +) + +SHAPE = (10, 9) +ARR = np.arange(90).reshape(SHAPE) + + +def _read_box(region): + """Simulate an engine read: the ndim-preserving box.""" + return ARR[tuple(slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True))] + + +@pytest.mark.parametrize( + "sel", + [ + (slice(2, 7), slice(0, 9)), + (slice(None), slice(3, 4)), + (slice(8, 2, -2), slice(None)), + (slice(1, 8, 3), slice(2, 9, 2)), + (3, slice(None)), + (-1, -2), + (Ellipsis, 4), + slice(5), + Ellipsis, + (slice(4, 4), slice(None)), # empty + ], +) +def test_normalize_basic_matches_numpy(sel) -> None: + region, post = normalize_basic(sel, SHAPE) + np.testing.assert_array_equal(_read_box(region)[post], ARR[sel]) + + +@pytest.mark.parametrize( + "sel", + [ + (np.array([1, 4, 7]), np.array([0, 8])), + (np.array([7, 1, 4]), slice(2, 6)), # unordered axis indices + (np.array([3, 3]), np.array([5, 5])), # repeats + (slice(1, 9, 2), np.array([2])), + (np.array([0]), 4), # int axis + ], +) +def test_normalize_orthogonal_matches_numpy_oindex(sel) -> None: + import numpy.lib.recfunctions # noqa: F401 (just ensures numpy fully loaded) + + region, post = normalize_orthogonal(sel, SHAPE) + # numpy oindex semantics == np.ix_ over per-axis index lists + axes = [] + for s, size in zip(sel if isinstance(sel, tuple) else (sel,), SHAPE, strict=False): + if isinstance(s, slice): + axes.append(np.arange(*s.indices(size))) + elif np.isscalar(s) or getattr(s, "ndim", 1) == 0: + axes.append(np.array([int(s)])) + else: + axes.append(np.asarray(s)) + expected = ARR[np.ix_(*axes)] + np.testing.assert_array_equal(_read_box(region)[post], expected) + + +def test_normalize_coordinate_matches_numpy_vindex() -> None: + coords = (np.array([9, 0, 3, 3]), np.array([8, 0, 2, 2])) + region, post = normalize_coordinate(coords, SHAPE) + np.testing.assert_array_equal(_read_box(region)[post], ARR[coords]) + + +def test_normalize_block_is_contiguous() -> None: + # chunk shape (3, 4) over SHAPE (10, 9): block (1, 2) spans rows 3:6, cols 8:9 + region = normalize_block((1, 2), chunk_grid_shape=(4, 3), chunk_shape=(3, 4), shape=SHAPE) + assert region.start == (3, 8) + assert region.end_exclusive == (6, 9) + + +def test_normalize_basic_rejects_fancy() -> None: + with pytest.raises(TypeError): + normalize_basic((np.array([1, 2]), slice(None)), SHAPE) + + +def test_normalize_basic_rejects_out_of_bounds_int() -> None: + with pytest.raises(IndexError): + normalize_basic((10, slice(None)), SHAPE) + + +def test_normalize_coordinate_rejects_out_of_bounds() -> None: + with pytest.raises(IndexError): + normalize_coordinate((np.array([10]), np.array([0])), SHAPE) + + +def test_normalize_orthogonal_rejects_boolean_ndim_mismatch() -> None: + with pytest.raises(IndexError): + normalize_orthogonal((np.zeros((2, 2), dtype=bool), slice(None)), SHAPE) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `uv run pytest tests/engine/test_normalize.py -v` +Expected: FAIL (`ImportError`) + +- [ ] **Step 3: Implement** + +```python +# src/zarr/core/engine/_normalize.py +"""Reduce public selection kinds to `(Region, post_index)` pairs. + +The engine boundary only speaks contiguous step-1 boxes (`Region`). Each +helper returns the box to transfer plus the numpy index that, applied to the +ndim-preserving box result, yields exactly `array[original_selection]`. +""" + +from __future__ import annotations + +import operator +import types +from typing import TYPE_CHECKING, Any + +import numpy as np + +from zarr.abc.engine import Region + +if TYPE_CHECKING: + from zarr.core.indexing import BasicSelection + + +def _expand(selection: Any, ndim: int) -> tuple[Any, ...]: + """Expand Ellipsis and pad missing trailing axes with full slices.""" + sel_tuple = selection if isinstance(selection, tuple) else (selection,) + n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) + if n_ellipsis > 1: + raise IndexError("an index can only have a single ellipsis ('...')") + if n_ellipsis == 1: + i = sel_tuple.index(Ellipsis) + n_fill = ndim - (len(sel_tuple) - 1) + if n_fill < 0: + raise IndexError(f"too many indices for array: array is {ndim}-dimensional") + sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] + if len(sel_tuple) > ndim: + raise IndexError(f"too many indices for array: array is {ndim}-dimensional") + return sel_tuple + (slice(None),) * (ndim - len(sel_tuple)) + + +def _normalize_int(sel: Any, size: int, dim: int) -> int: + idx = operator.index(sel) + if idx < 0: + idx += size + if not 0 <= idx < size: + raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") + return idx + + +def normalize_basic( + selection: BasicSelection, shape: tuple[int, ...] +) -> tuple[Region, tuple[slice | int, ...]]: + """Normalize a numpy basic-indexing selection to a step-1 box. + + Supports integers, slices (any step), and `Ellipsis`. Integer axes get a + length-1 range in the box and `0` in the post index (dropping the axis, + matching numpy). Fancy elements raise `TypeError`. + """ + sel_tuple = _expand(selection, len(shape)) + starts: list[int] = [] + ends: list[int] = [] + post: list[slice | int] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + start, stop, step = sel.indices(size) + n = len(range(start, stop, step)) + if n == 0: + starts.append(0) + ends.append(0) + post.append(slice(None)) + elif step > 0: + starts.append(start) + ends.append(start + (n - 1) * step + 1) + post.append(slice(None, None, step)) + else: + last = start + (n - 1) * step + starts.append(last) + ends.append(start + 1) + post.append(slice(None, None, step)) + elif isinstance(sel, types.EllipsisType): + raise AssertionError("Ellipsis expanded by _expand") + else: + try: + idx = _normalize_int(sel, size, dim) + except TypeError: + raise TypeError( + f"unsupported selection element {sel!r}: only integers, " + "slices, and Ellipsis are supported by basic indexing" + ) from None + starts.append(idx) + ends.append(idx + 1) + post.append(0) + return Region(start=tuple(starts), end_exclusive=tuple(ends)), tuple(post) + + +def normalize_orthogonal( + selection: Any, shape: tuple[int, ...] +) -> tuple[Region, tuple[Any, ...]]: + """Normalize an orthogonal (`oindex`) selection to a box + outer index. + + Each axis selector may be an integer, a slice, an integer array, or a 1-d + boolean mask for that axis. The post index is the `np.ix_`-broadcastable + tuple of per-axis integer arrays (relative to the box origin), with + integer axes dropped afterwards by the facade via the returned index + (integers appear as scalar entries produced by `np.ix_` inputs of shape + `(1,)` — the facade result keeps numpy `oindex` semantics). + """ + sel_tuple = _expand(selection, len(shape)) + starts: list[int] = [] + ends: list[int] = [] + axis_indices: list[np.ndarray] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + idxs = np.arange(*sel.indices(size)) + elif isinstance(sel, np.ndarray) and sel.dtype == bool: + if sel.ndim != 1 or sel.shape[0] != size: + raise IndexError( + f"boolean index for axis {dim} must be 1-d with length {size}" + ) + idxs = np.nonzero(sel)[0] + elif isinstance(sel, (np.ndarray, list)): + idxs = np.asarray(sel, dtype=np.intp) + if idxs.ndim != 1: + raise IndexError(f"orthogonal index for axis {dim} must be 1-d") + idxs = np.where(idxs < 0, idxs + size, idxs) + if idxs.size and (idxs.min() < 0 or idxs.max() >= size): + raise IndexError(f"index out of bounds for axis {dim} with size {size}") + else: + idxs = np.array([_normalize_int(sel, size, dim)], dtype=np.intp) + if idxs.size == 0: + starts.append(0) + ends.append(0) + axis_indices.append(idxs) + else: + lo, hi = int(idxs.min()), int(idxs.max()) + starts.append(lo) + ends.append(hi + 1) + axis_indices.append(idxs - lo) + region = Region(start=tuple(starts), end_exclusive=tuple(ends)) + post = np.ix_(*axis_indices) if axis_indices else () + # integer axes were widened to length-1 arrays; the caller squeezes them + squeeze_axes = tuple( + i + for i, sel in enumerate(sel_tuple) + if not isinstance(sel, (slice, np.ndarray, list)) + ) + if squeeze_axes: + return region, (*post, _Squeeze(squeeze_axes)) # type: ignore[return-value] + return region, post + + +class _Squeeze: + """Marker appended to an orthogonal post index: squeeze these axes.""" + + def __init__(self, axes: tuple[int, ...]) -> None: + self.axes = axes + + +def apply_post_index(box: np.ndarray, post: tuple[Any, ...]) -> np.ndarray: + """Apply a post index produced by a `normalize_*` helper to a box read.""" + if post and isinstance(post[-1], _Squeeze): + result = box[post[:-1]] if len(post) > 1 else box + return np.squeeze(result, axis=post[-1].axes) + if post == (): + return box + return box[post] + + +def normalize_coordinate( + selection: tuple[Any, ...], shape: tuple[int, ...] +) -> tuple[Region, tuple[np.ndarray, ...]]: + """Normalize a coordinate (`vindex`) selection to a box + pointwise index.""" + coords = tuple(np.asarray(c, dtype=np.intp) for c in selection) + if len(coords) != len(shape): + raise IndexError( + f"coordinate selection needs {len(shape)} axis arrays, got {len(coords)}" + ) + coords = tuple( + np.where(c < 0, c + size, c) for c, size in zip(coords, shape, strict=True) + ) + for dim, (c, size) in enumerate(zip(coords, shape, strict=True)): + if c.size and (c.min() < 0 or c.max() >= size): + raise IndexError(f"index out of bounds for axis {dim} with size {size}") + starts = tuple(int(c.min()) if c.size else 0 for c in coords) + ends = tuple(int(c.max()) + 1 if c.size else 0 for c in coords) + post = tuple(c - s for c, s in zip(coords, starts, strict=True)) + return Region(start=starts, end_exclusive=ends), post + + +def normalize_block( + block_coords: tuple[int, ...], + *, + chunk_grid_shape: tuple[int, ...], + chunk_shape: tuple[int, ...], + shape: tuple[int, ...], +) -> Region: + """Normalize a block (chunk-grid) selection to its contiguous box.""" + starts = [] + ends = [] + for dim, (b, nblocks, csize, size) in enumerate( + zip(block_coords, chunk_grid_shape, chunk_shape, shape, strict=True) + ): + idx = _normalize_int(b, nblocks, dim) + starts.append(idx * csize) + ends.append(min((idx + 1) * csize, size)) + return Region(start=tuple(starts), end_exclusive=tuple(ends)) +``` + +```python +# src/zarr/core/engine/__init__.py +from zarr.core.engine._normalize import ( + apply_post_index, + normalize_basic, + normalize_block, + normalize_coordinate, + normalize_orthogonal, +) + +__all__ = [ + "apply_post_index", + "normalize_basic", + "normalize_block", + "normalize_coordinate", + "normalize_orthogonal", +] +``` + +Note for the implementer: the test uses `_read_box(region)[post]` for basic and +coordinate kinds and `apply_post_index` semantics for orthogonal. Update the +orthogonal test to call `apply_post_index(_read_box(region), post)` — plain +`box[post]` is wrong once the `_Squeeze` marker is present. + +- [ ] **Step 4: Run tests** + +Run: `uv run pytest tests/engine/test_normalize.py -v` — Expected: PASS +Run: `uv run mypy src/zarr/core/engine` — Expected: no errors + +- [ ] **Step 5: Commit** + +```bash +git add src/zarr/core/engine tests/engine/test_normalize.py +git commit -m "feat: normalize selection kinds to Region + post-index" +``` + +--- + +### Task 3: Default engines (`_default.py`) + +**Files:** +- Create: `src/zarr/core/engine/_default.py` +- Modify: `src/zarr/core/engine/__init__.py` (re-export) +- Test: `tests/engine/test_default_engine.py` + +**Interfaces:** +- Consumes: `Region` (Task 1). +- Produces: + - `DefaultAsyncArrayEngine(store_path: StorePath, metadata: ArrayMetadata, config: ArrayConfig)` satisfying `AsyncArrayEngine`. + - `DefaultArrayEngine(async_engine: DefaultAsyncArrayEngine)` satisfying `ArrayEngine` (wraps with `zarr.core.sync.sync`). + - `DefaultAsyncHierarchyEngine(store: Store)` / `DefaultHierarchyEngine(store: Store)` with `array_engine(path, metadata)`. +- Task 5 constructs these inside `AsyncArray.__init__`/`Array`. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/engine/test_default_engine.py +from __future__ import annotations + +import numpy as np + +import zarr +from zarr.abc.engine import AsyncArrayEngine, Region +from zarr.core.buffer import default_buffer_prototype +from zarr.core.engine import DefaultArrayEngine, DefaultAsyncArrayEngine +from zarr.core.sync import sync +from zarr.storage import MemoryStore + + +def _make_array() -> zarr.Array: + z = zarr.create_array( + MemoryStore(), shape=(10, 9), chunks=(3, 4), dtype="int32", fill_value=0 + ) + z[:, :] = np.arange(90, dtype="int32").reshape(10, 9) + return z + + +def test_default_async_engine_read_write_roundtrip() -> None: + z = _make_array() + eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + assert isinstance(eng, AsyncArrayEngine) + proto = default_buffer_prototype() + region = Region(start=(2, 1), end_exclusive=(7, 5)) + + out = sync(eng.read_selection(region, prototype=proto)) + np.testing.assert_array_equal(np.asarray(out), np.asarray(z[2:7, 1:5])) + + new = np.full((5, 4), -1, dtype="int32") + value = proto.nd_buffer.from_ndarray_like(new) + sync(eng.write_selection(region, value, prototype=proto)) + np.testing.assert_array_equal(np.asarray(z[2:7, 1:5]), new) + + +def test_default_sync_engine_matches_async() -> None: + z = _make_array() + async_eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + eng = DefaultArrayEngine(async_eng) + proto = default_buffer_prototype() + region = Region(start=(0, 0), end_exclusive=(10, 9)) + np.testing.assert_array_equal( + np.asarray(eng.read_selection(region, prototype=proto)), np.asarray(z[:, :]) + ) + + +def test_with_metadata_rebinds() -> None: + z = _make_array() + eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + new_meta = z.async_array.metadata + assert eng.with_metadata(new_meta) is not eng +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `uv run pytest tests/engine/test_default_engine.py -v` +Expected: FAIL (`ImportError: DefaultArrayEngine`) + +- [ ] **Step 3: Implement** + +```python +# src/zarr/core/engine/_default.py +"""The default engines: today's codec-pipeline machinery behind the protocol.""" + +from __future__ import annotations + +from dataclasses import replace +from typing import TYPE_CHECKING + +from zarr.abc.engine import Region +from zarr.core.array_spec import ArraySpec +from zarr.core.chunk_grids import ChunkGrid +from zarr.core.codec_pipeline import create_codec_pipeline +from zarr.core.indexing import BasicIndexer +from zarr.core.metadata import ArrayMetadata +from zarr.core.sync import sync + +if TYPE_CHECKING: + from zarr.abc.store import Store + from zarr.core.array_spec import ArrayConfig + from zarr.core.buffer import BufferPrototype, NDBuffer + from zarr.core.common import NDArrayLike + from zarr.core.storage import StorePath + + +def _region_to_slices(region: Region) -> tuple[slice, ...]: + return tuple( + slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True) + ) + + +class DefaultAsyncArrayEngine: + """Codec-pipeline-backed engine. Any store, Zarr v2 and v3.""" + + def __init__( + self, store_path: StorePath, metadata: ArrayMetadata, config: ArrayConfig + ) -> None: + self.store_path = store_path + self.metadata = metadata + self.config = config + self._chunk_grid = ChunkGrid.from_metadata(metadata) + self._codec_pipeline = create_codec_pipeline( + metadata=metadata, store=store_path.store + ) + + def with_metadata(self, metadata: ArrayMetadata) -> DefaultAsyncArrayEngine: + return DefaultAsyncArrayEngine( + store_path=self.store_path, metadata=metadata, config=self.config + ) + + def _indexer(self, region: Region) -> BasicIndexer: + return BasicIndexer( + _region_to_slices(region), + shape=self.metadata.shape, + chunk_grid=self._chunk_grid, + ) + + def _chunk_spec(self, prototype: BufferPrototype) -> tuple[ArraySpec | None, ArrayConfig]: + # v2 honors metadata order; v3 honors config order (mirrors _get_selection) + config = self.config + if self.metadata.zarr_format == 2: + config = replace(config, order=self.metadata.order) + if self._chunk_grid.is_regular: + return ( + ArraySpec( + shape=self._chunk_grid.chunk_shape, + dtype=self.metadata.dtype, + fill_value=self.metadata.fill_value, + config=config, + prototype=prototype, + ), + config, + ) + return None, config + + async def read_selection( + self, selection: Region, *, prototype: BufferPrototype + ) -> NDArrayLike: + indexer = self._indexer(selection) + if self.metadata.zarr_format == 2: + dtype = self.metadata.dtype.to_native_dtype() + order = self.metadata.order + else: + dtype = self.metadata.data_type.to_native_dtype() + order = self.config.order + out_buffer = prototype.nd_buffer.empty( + shape=indexer.shape, dtype=dtype, order=order + ) + if all(e > s for s, e in zip(selection.start, selection.end_exclusive, strict=True)): + spec, config = self._chunk_spec(prototype) + await self._codec_pipeline.read( + [ + ( + self.store_path / self.metadata.encode_chunk_key(chunk_coords), + spec + if spec is not None + else _irregular_chunk_spec( + self.metadata, self._chunk_grid, chunk_coords, config, prototype + ), + chunk_selection, + out_selection, + is_complete_chunk, + ) + for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexer + ], + out_buffer, + drop_axes=indexer.drop_axes, + ) + return out_buffer.as_ndarray_like() + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + indexer = self._indexer(selection) + spec, config = self._chunk_spec(prototype) + await self._codec_pipeline.write( + [ + ( + self.store_path / self.metadata.encode_chunk_key(chunk_coords), + spec + if spec is not None + else _irregular_chunk_spec( + self.metadata, self._chunk_grid, chunk_coords, config, prototype + ), + chunk_selection, + out_selection, + is_complete_chunk, + ) + for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexer + ], + value, + drop_axes=indexer.drop_axes, + ) + + +def _irregular_chunk_spec(metadata, chunk_grid, chunk_coords, config, prototype): + # same construction as array.py::_get_chunk_spec — import and delegate + from zarr.core.array import _get_chunk_spec + + return _get_chunk_spec(metadata, chunk_grid, chunk_coords, config, prototype) + + +class DefaultArrayEngine: + """Sync adapter over `DefaultAsyncArrayEngine` via `sync()`.""" + + def __init__(self, async_engine: DefaultAsyncArrayEngine) -> None: + self._async = async_engine + + def read_selection( + self, selection: Region, *, prototype: BufferPrototype + ) -> NDArrayLike: + return sync(self._async.read_selection(selection, prototype=prototype)) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + sync(self._async.write_selection(selection, value, prototype=prototype)) + + def with_metadata(self, metadata: ArrayMetadata) -> DefaultArrayEngine: + return DefaultArrayEngine(self._async.with_metadata(metadata)) + + +class DefaultAsyncHierarchyEngine: + """Store-bound factory for default async engines.""" + + def __init__(self, store: Store) -> None: + self.store = store + + def array_engine(self, path: str, metadata: ArrayMetadata) -> DefaultAsyncArrayEngine: + from zarr.core.storage import StorePath + + from zarr.core.array_spec import parse_array_config + + return DefaultAsyncArrayEngine( + store_path=StorePath(self.store, path), + metadata=metadata, + config=parse_array_config(None), + ) + + +class DefaultHierarchyEngine: + """Store-bound factory for default sync engines.""" + + def __init__(self, store: Store) -> None: + self._async = DefaultAsyncHierarchyEngine(store) + + def array_engine(self, path: str, metadata: ArrayMetadata) -> DefaultArrayEngine: + return DefaultArrayEngine(self._async.array_engine(path, metadata)) +``` + +Implementation notes (verify against the live code, these matter): +- `StorePath` import path: check `src/zarr/core/array.py`'s own import and copy it. +- The empty-region guard in `read_selection` mirrors `_get_selection`'s + `product(indexer.shape) > 0` check; use `zarr.core.common.product` if simpler. +- v2 `metadata.dtype` / v3 `metadata.data_type` split is copied from + `_get_selection` (array.py:5392-5397). Do not invent a new path. + +Add to `src/zarr/core/engine/__init__.py`: + +```python +from zarr.core.engine._default import ( + DefaultArrayEngine, + DefaultAsyncArrayEngine, + DefaultAsyncHierarchyEngine, + DefaultHierarchyEngine, +) +``` +(and extend `__all__` accordingly). + +- [ ] **Step 4: Run tests** + +Run: `uv run pytest tests/engine/test_default_engine.py tests/engine/test_protocols.py -v` +Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add src/zarr/core/engine tests/engine/test_default_engine.py +git commit -m "feat: default array engines over the codec pipeline" +``` + +--- + +### Task 4: Engine resolution (`_resolve.py`) + +**Files:** +- Create: `src/zarr/core/engine/_resolve.py` +- Modify: `src/zarr/core/engine/__init__.py` (re-export) +- Test: `tests/engine/test_resolve.py` + +**Interfaces:** +- Consumes: default engines (Task 3), protocols (Task 1). +- Produces (used by Task 5's `AsyncArray.__init__` / `Array` wiring): + - `EngineName = Literal["default", "zarrista"]` + - `resolve_async_engine(engine: AsyncArrayEngine | EngineName | None, *, store: Store, path: str, metadata: ArrayMetadata) -> AsyncArrayEngine` + - `resolve_sync_engine(engine: ArrayEngine | EngineName | None, *, store: Store, path: str, metadata: ArrayMetadata) -> ArrayEngine` + - `None` and `"default"` → default engine; `"zarrista"` → lazy import of `zarr.zarrista`, raising a clear `ImportError` if zarrista is missing; an instance → returned as-is. + - Hierarchy engines are cached per `(name, id(store))` in a module-level `WeakValueDictionary`-free plain dict keyed by a `weakref.ref` of the store (evicted via callback) so engines from one store share resources. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/engine/test_resolve.py +from __future__ import annotations + +import numpy as np +import pytest + +import zarr +from zarr.core.engine import ( + DefaultArrayEngine, + DefaultAsyncArrayEngine, + resolve_async_engine, + resolve_sync_engine, +) +from zarr.storage import MemoryStore + + +def _array() -> zarr.Array: + return zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + + +def test_resolution_combinations() -> None: + z = _array() + store = z.store + path = z.path + meta = z.async_array.metadata + + # None and "default" produce default engines + for spec in (None, "default"): + assert isinstance( + resolve_async_engine(spec, store=store, path=path, metadata=meta), + DefaultAsyncArrayEngine, + ) + assert isinstance( + resolve_sync_engine(spec, store=store, path=path, metadata=meta), + DefaultArrayEngine, + ) + + # instances pass through untouched + inst = resolve_sync_engine(None, store=store, path=path, metadata=meta) + assert resolve_sync_engine(inst, store=store, path=path, metadata=meta) is inst + + # engines minted from the same store share a hierarchy engine + e1 = resolve_async_engine(None, store=store, path=path, metadata=meta) + e2 = resolve_async_engine(None, store=store, path="other", metadata=meta) + assert e1.store_path.store is e2.store_path.store + + +def test_unknown_name_raises() -> None: + z = _array() + with pytest.raises(ValueError, match="unknown engine"): + resolve_async_engine( + "bogus", # type: ignore[arg-type] + store=z.store, + path=z.path, + metadata=z.async_array.metadata, + ) + + +def test_zarrista_missing_raises_import_error() -> None: + pytest.importorskip("zarr.zarrista", reason="run only when zarrista absent") \ + if False else None + try: + import zarrista # noqa: F401 + + pytest.skip("zarrista installed; missing-module error not testable") + except ImportError: + pass + z = _array() + with pytest.raises(ImportError, match="zarrista"): + resolve_async_engine( + "zarrista", store=z.store, path=z.path, metadata=z.async_array.metadata + ) +``` + +- [ ] **Step 2: Run to verify failure** + +Run: `uv run pytest tests/engine/test_resolve.py -v` +Expected: FAIL (`ImportError: resolve_async_engine`) + +- [ ] **Step 3: Implement** + +```python +# src/zarr/core/engine/_resolve.py +"""Resolve an `engine=` argument to a bound array engine.""" + +from __future__ import annotations + +import weakref +from typing import TYPE_CHECKING, Literal + +from zarr.core.engine._default import ( + DefaultAsyncHierarchyEngine, + DefaultHierarchyEngine, +) + +if TYPE_CHECKING: + from zarr.abc.engine import ( + ArrayEngine, + AsyncArrayEngine, + AsyncHierarchyEngine, + HierarchyEngine, + ) + from zarr.abc.store import Store + from zarr.core.metadata import ArrayMetadata + +EngineName = Literal["default", "zarrista"] + +# (name, kind, id(store)) -> hierarchy engine; entries evicted when the store dies +_hierarchy_cache: dict[tuple[str, str, int], object] = {} + + +def _cached_hierarchy(name: str, kind: str, store: Store, factory) -> object: + key = (name, kind, id(store)) + if key not in _hierarchy_cache: + _hierarchy_cache[key] = factory(store) + weakref.finalize(store, _hierarchy_cache.pop, key, None) + return _hierarchy_cache[key] + + +def _hierarchy_factory(name: str, *, sync: bool): + if name == "default": + return DefaultHierarchyEngine if sync else DefaultAsyncHierarchyEngine + if name == "zarrista": + try: + from zarr.zarrista import ( + ZarristaAsyncHierarchyEngine, + ZarristaHierarchyEngine, + ) + except ImportError as e: + raise ImportError( + "engine='zarrista' requires the `zarrista` package; " + "install zarr with the `zarrista` extra" + ) from e + return ZarristaHierarchyEngine if sync else ZarristaAsyncHierarchyEngine + raise ValueError(f"unknown engine name {name!r}; expected 'default' or 'zarrista'") + + +def resolve_async_engine( + engine: AsyncArrayEngine | EngineName | None, + *, + store: Store, + path: str, + metadata: ArrayMetadata, +) -> AsyncArrayEngine: + if engine is None: + engine = "default" + if isinstance(engine, str): + factory = _hierarchy_factory(engine, sync=False) + hierarchy: AsyncHierarchyEngine = _cached_hierarchy( # type: ignore[assignment] + engine, "async", store, factory + ) + return hierarchy.array_engine(path, metadata) + return engine + + +def resolve_sync_engine( + engine: ArrayEngine | EngineName | None, + *, + store: Store, + path: str, + metadata: ArrayMetadata, +) -> ArrayEngine: + if engine is None: + engine = "default" + if isinstance(engine, str): + factory = _hierarchy_factory(engine, sync=True) + hierarchy: HierarchyEngine = _cached_hierarchy( # type: ignore[assignment] + engine, "sync", store, factory + ) + return hierarchy.array_engine(path, metadata) + return engine +``` + +Note: if `weakref.finalize` on a `Store` fails (stores with `__slots__` and no +`__weakref__`), fall back to a `weakref.WeakKeyDictionary[Store, dict[str, object]]` +keyed on the store; adjust the test only if the sharing assertion still holds. + +Re-export `EngineName`, `resolve_async_engine`, `resolve_sync_engine` from +`zarr.core.engine.__init__` and extend `__all__`. + +- [ ] **Step 4: Run tests** + +Run: `uv run pytest tests/engine/test_resolve.py -v` — Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add src/zarr/core/engine tests/engine/test_resolve.py +git commit -m "feat: engine-spec resolution with per-store hierarchy cache" +``` + +--- + +### Task 5: Wire `AsyncArray` through its engine + +The public behavior contract for this task and Task 6: **the whole existing +array API test suite keeps passing** (`tests/test_array.py`, `tests/test_indexing.py`). +That suite is the real acceptance test; the new unit tests below only pin the +wiring itself. + +**Files:** +- Modify: `src/zarr/core/array.py` +- Test: `tests/engine/test_asyncarray_wiring.py` + +**Interfaces:** +- Consumes: `resolve_async_engine` (Task 4), `normalize_*`/`apply_post_index` (Task 2). +- Produces: `AsyncArray.__init__(..., engine: AsyncArrayEngine | EngineName | None = None)`; attribute `AsyncArray.engine`; rewritten module-level `_get_selection(engine, indexer_or_selection…)`/`_set_selection` (final signatures below). Task 6 mirrors this for `Array`; Task 7 threads `engine=` through creation APIs. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/engine/test_asyncarray_wiring.py +from __future__ import annotations + +import numpy as np + +import zarr +from zarr.abc.engine import Region +from zarr.core.engine import DefaultAsyncArrayEngine +from zarr.storage import MemoryStore + + +class _SpyEngine: + """Wraps a real engine, recording regions.""" + + def __init__(self, inner: DefaultAsyncArrayEngine) -> None: + self.inner = inner + self.read_regions: list[Region] = [] + self.write_regions: list[Region] = [] + + async def read_selection(self, selection, *, prototype): + self.read_regions.append(selection) + return await self.inner.read_selection(selection, prototype=prototype) + + async def write_selection(self, selection, value, *, prototype): + self.write_regions.append(selection) + return await self.inner.write_selection(selection, value, prototype=prototype) + + def with_metadata(self, metadata): + return _SpyEngine(self.inner.with_metadata(metadata)) + + +def test_asyncarray_routes_io_through_engine() -> None: + z = zarr.create_array(MemoryStore(), shape=(10,), chunks=(3,), dtype="int16") + aa = z.async_array + spy = _SpyEngine( + DefaultAsyncArrayEngine( + store_path=aa.store_path, metadata=aa.metadata, config=aa.config + ) + ) + object.__setattr__(aa, "engine", spy) + + z[2:8] = np.arange(6, dtype="int16") + data = z[2:8] + + assert spy.write_regions == [Region(start=(2,), end_exclusive=(8,))] + assert spy.read_regions == [Region(start=(2,), end_exclusive=(8,))] + np.testing.assert_array_equal(np.asarray(data), np.arange(6, dtype="int16")) + + +def test_asyncarray_default_engine_attribute() -> None: + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + assert isinstance(z.async_array.engine, DefaultAsyncArrayEngine) +``` + +Note: `test_asyncarray_routes_io_through_engine` depends on Task 6 (the sync +`Array` path shares the async engine attribute until Task 6 gives `Array` its +own sync engine; when implementing Task 5 alone, exercise `await aa.getitem(...)` +instead of `z[...]` — final test content above is what must pass after Task 6). + +- [ ] **Step 2: Implement the `AsyncArray` changes** + +In `src/zarr/core/array.py`: + +1. `AsyncArray.__init__` (array.py:349): add keyword + `engine: AsyncArrayEngine | EngineName | None = None` to all three + signatures (both overloads + implementation), and set + +```python +object.__setattr__( + self, + "engine", + resolve_async_engine( + engine, + store=store_path.store, + path=store_path.path, + metadata=metadata_parsed, + ), +) +``` + + with attribute declaration `engine: AsyncArrayEngine = field(init=False)` + next to `codec_pipeline` (array.py:329). Import `resolve_async_engine` + and `EngineName` from `zarr.core.engine`, `AsyncArrayEngine` from + `zarr.abc.engine` (TYPE_CHECKING block for the types). + +2. Metadata-mutating paths: wherever `AsyncArray` constructs a successor + `AsyncArray` with new metadata (`resize`, `append`, `update_attributes`, + `with_config` — find them with + `grep -n "type(self)(" src/zarr/core/array.py` and + `grep -n "replace(self" src/zarr/core/array.py`), pass + `engine=self.engine.with_metadata(new_metadata)` if the constructor + accepts it, or call `object.__setattr__(new_array, "engine", + self.engine.with_metadata(new_metadata))` right after construction. + +3. Rewrite the two internal I/O helpers. Replace the bodies of + `AsyncArray._get_selection` (array.py:1412) and + `AsyncArray._set_selection` (array.py:1550) and the module-level + `_get_selection` (array.py:5353) / `_set_selection` (array.py:5702) with + engine-routed versions. The module-level functions get new signatures + (all callers are inside this file and the sync `Array` methods — + update every call site found via + `grep -n "_get_selection\|_set_selection" src/zarr/core/array.py`): + +```python +async def _get_selection( + engine: AsyncArrayEngine, + metadata: ArrayMetadata, + config: ArrayConfig, + region: Region, + post_index: tuple[Any, ...], + *, + prototype: BufferPrototype, + out: NDBuffer | None = None, + fields: Fields | None = None, +) -> NDArrayLikeOrScalar: + dtype = ( + metadata.dtype if metadata.zarr_format == 2 else metadata.data_type + ).to_native_dtype() + out_dtype = check_fields(fields, dtype) + if product(region.shape) == 0: + empty = np.empty( + apply_post_index(np.empty(region.shape, dtype=out_dtype), post_index).shape, + dtype=out_dtype, + ) + return _finalize_result(empty, out) + box = np.asarray(await engine.read_selection(region, prototype=prototype)) + if fields is not None: + box = box[fields] + result = apply_post_index(box, post_index) + return _finalize_result(result, out) + + +def _finalize_result(result: np.ndarray, out: NDBuffer | None) -> NDArrayLikeOrScalar: + if out is not None: + out.as_ndarray_like()[...] = result + return out.as_ndarray_like() + if result.shape == (): + return result[()] # scalar extraction, matching current behavior + return result +``` + +```python +async def _set_selection( + engine: AsyncArrayEngine, + metadata: ArrayMetadata, + config: ArrayConfig, + region: Region, + post_index: tuple[Any, ...], + value: npt.ArrayLike, + *, + prototype: BufferPrototype, + fields: Fields | None = None, +) -> None: + dtype = ( + metadata.dtype if metadata.zarr_format == 2 else metadata.data_type + ).to_native_dtype() + check_fields(fields, dtype) + fields = check_no_multi_fields(fields) + if product(region.shape) == 0: + return + value_np = np.asanyarray(value, dtype=None if fields else dtype) + + identity_post = all( + isinstance(p, slice) and p == slice(None) for p in post_index + ) and fields is None + if identity_post: + # broadcast the value to the full ndim-preserving box + box = np.broadcast_to(value_np, region.shape).astype(dtype, copy=False) + else: + # facade-level read-modify-write for strided/fancy/fields writes + box = np.array( + np.asarray(await engine.read_selection(region, prototype=prototype)) + ) + target = box[fields] if fields is not None else box + if post_index == (): + target[...] = value_np + else: + # _Squeeze markers only occur on reads (orthogonal); writes use + # the raw np.ix_/pointwise index without the marker + target[post_index] = value_np + value_buffer = prototype.nd_buffer.from_ndarray_like( + np.ascontiguousarray(box) + ) + await engine.write_selection(region, value_buffer, prototype=prototype) +``` + + Implementation notes: + - Keep the existing scalar/array-like `value` coercion logic from the old + `_set_selection` (array.py:5749-5769) ahead of the code above where it is + richer than `np.asanyarray` (non-numpy array types); preserve it rather + than the simplified line if any existing test fails. + - Writes for orthogonal selections must pass the *raw* per-axis outer index + (`np.ix_(...)` result without the `_Squeeze` marker); add a + `strip_squeeze(post) -> post` helper in `_normalize.py` and use it here. + - The old `regular_chunk_spec` optimization lives in the default engine + now (Task 3); nothing here touches chunks. + +4. Update `AsyncArray` methods that build indexers + (`getitem`, `setitem`, `get_basic_selection`, `set_basic_selection`, + `get_orthogonal_selection`, `set_orthogonal_selection`, + `get_mask_selection`, `set_mask_selection`, + `get_coordinate_selection`, `set_coordinate_selection`, + `get_block_selection`, `set_block_selection` — find each with + `grep -n "Indexer(" src/zarr/core/array.py`). Replace + `indexer = BasicIndexer(...)` + `self._get_selection(indexer, ...)` with: + +```python +region, post = normalize_basic(selection, self.shape) +return await _get_selection( + self.engine, self.metadata, self.config, region, post, + prototype=prototype, out=out, fields=fields, +) +``` + + and analogously `normalize_orthogonal` for orthogonal, `normalize_coordinate` + for coordinate, `np.nonzero(mask)` → `normalize_coordinate` for mask, and + `normalize_block(...)` (post index `()`) for block selections, passing the + chunk-grid shape from `self._chunk_grid`. Mask/coordinate *reads* return the + pointwise result (already numpy `vindex` semantics via the pointwise post + index). Delete the now-unused `Indexer` imports only if nothing else in the + file uses them (resize/`nchunks_initialized` may). + +- [ ] **Step 3: Run the wiring test and the array suites** + +Run: `uv run pytest tests/engine/test_asyncarray_wiring.py -v` — Expected: PASS (the async variant of the spy test if Task 6 not yet done) +Run: `uv run pytest tests/test_array.py tests/test_indexing.py -x -q` — Expected: PASS. Any failure here is a behavior regression: fix it before proceeding, favoring the old code's semantics. + +- [ ] **Step 4: Commit** + +```bash +git add src/zarr/core/array.py src/zarr/core/engine tests/engine/test_asyncarray_wiring.py +git commit -m "feat: AsyncArray routes selection I/O through its engine" +``` + +--- + +### Task 6: Sync `Array` data path (no event loop) + +**Files:** +- Modify: `src/zarr/core/array.py` +- Test: `tests/engine/test_sync_path.py` + +**Interfaces:** +- Consumes: `resolve_sync_engine` (Task 4), normalization helpers (Task 2), the `_finalize_result`/RMW patterns from Task 5. +- Produces: `Array.engine: ArrayEngine` property; sync module-level `_get_selection_sync`/`_set_selection_sync` mirroring Task 5's helpers with `engine.read_selection(...)` (no await). `Array.__init__`/constructors accept `engine: ArrayEngine | EngineName | None = None`. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/engine/test_sync_path.py +from __future__ import annotations + +import asyncio + +import numpy as np +import pytest + +import zarr +from zarr.abc.engine import ArrayEngine, Region +from zarr.core.engine import DefaultArrayEngine +from zarr.storage import MemoryStore + + +def test_array_has_sync_engine() -> None: + z = zarr.create_array(MemoryStore(), shape=(6,), chunks=(2,), dtype="uint8") + assert isinstance(z.engine, ArrayEngine) + assert isinstance(z.engine, DefaultArrayEngine) + + +class _NoLoopEngine: + """A sync engine that asserts no event loop is running when called.""" + + def __init__(self, inner) -> None: + self._inner = inner + self.calls = 0 + + def read_selection(self, selection, *, prototype): + self.calls += 1 + with pytest.raises(RuntimeError): + asyncio.get_running_loop() + return self._inner.read_selection(selection, prototype=prototype) + + def write_selection(self, selection, value, *, prototype): + self.calls += 1 + with pytest.raises(RuntimeError): + asyncio.get_running_loop() + return self._inner.write_selection(selection, value, prototype=prototype) + + def with_metadata(self, metadata): + return _NoLoopEngine(self._inner.with_metadata(metadata)) + + +def test_sync_data_path_runs_without_event_loop_in_caller_thread() -> None: + z = zarr.create_array(MemoryStore(), shape=(6,), chunks=(2,), dtype="uint8") + probe = _NoLoopEngine(z.engine) + object.__setattr__(z, "_engine", probe) # match the attribute name used in impl + + z[1:5] = np.arange(4, dtype="uint8") + out = z[1:5] + + assert probe.calls == 2 + np.testing.assert_array_equal(np.asarray(out), np.arange(4, dtype="uint8")) +``` + +(The engine methods themselves run in the caller thread; the *default* engine +internally uses `sync()` which runs coroutines on zarr's loop thread — that is +allowed. The probe asserts the caller thread has no running loop, i.e. `Array` +did not wrap the engine call in a coroutine.) + +- [ ] **Step 2: Implement** + +1. `Array` gains a lazily-resolved sync engine: store the creation-time spec + and resolve on first use. + +```python +# on class Array (near _async_array declaration, array.py:1803) +_engine: ArrayEngine | None = None +_engine_spec: ArrayEngine | EngineName | None = None + +@property +def engine(self) -> ArrayEngine: + """The synchronous array engine serving this array's data path.""" + if self._engine is None: + aa = self._async_array + object.__setattr__( + self, + "_engine", + resolve_sync_engine( + self._engine_spec, + store=aa.store_path.store, + path=aa.store_path.path, + metadata=aa.metadata, + ), + ) + object.__setattr__(self, "_engine_spec", None) + return self._engine +``` + + Whatever constructor `Array` uses (dataclass or `__init__` — check + `grep -n "def __init__\|@dataclass" src/zarr/core/array.py` around line + 1798) gains the optional `engine` keyword storing `_engine_spec`. When + `Array` wraps an `AsyncArray` that was itself given a *name* spec, reuse + that name so sync and async engines come from the same family. + +2. Every sync data method that currently does + `sync(self.async_array._get_selection(indexer, ...))` + (sites: array.py:2830, 2939, 3068, 3186, 3274, 3362, 3451, 3563, 3664, 3763 + — re-grep, they will have shifted after Task 5) is rewritten to the sync + mirror: + +```python +region, post = normalize_basic(selection, self.shape) # kind-appropriate helper +return _get_selection_sync( + self.engine, self._async_array.metadata, self._async_array.config, + region, post, prototype=prototype, out=out, fields=fields, +) +``` + +3. Add module-level `_get_selection_sync` / `_set_selection_sync` beside the + async versions from Task 5 — same bodies with `engine.read_selection(...)` / + `engine.write_selection(...)` un-awaited. Factor the shared pure logic + (dtype/fields resolution, empty short-circuit, box patching, result + finalization) into small helper functions used by both so the async and + sync variants differ only in the two engine calls. + +4. Metadata mutation on `Array` (`resize`, `append`) invalidates the cached + sync engine: `object.__setattr__(self, "_engine", self._engine.with_metadata(new_meta))` + when `_engine` is not None (sites via `grep -n "def resize\|def append" src/zarr/core/array.py`). + +- [ ] **Step 3: Run tests** + +Run: `uv run pytest tests/engine/ -v` — Expected: PASS (including the full Task 5 spy test now) +Run: `uv run pytest tests/test_array.py tests/test_indexing.py -q` — Expected: PASS +Run: `uv run mypy src/zarr` — Expected: no new errors + +- [ ] **Step 4: Commit** + +```bash +git add src/zarr/core/array.py src/zarr/core/engine tests/engine/test_sync_path.py +git commit -m "feat: Array sync data path calls its engine without the event loop" +``` + +--- + +### Task 7: `engine=` on creation/open APIs + +**Files:** +- Modify: `src/zarr/core/array.py` (`AsyncArray.open`, `Array.open`, `_create` paths), `src/zarr/api/asynchronous.py` and `src/zarr/api/synchronous.py` (`create_array`, `open_array`), `src/zarr/core/array_creation.py` if `create_array` lives there (find with `grep -rn "def create_array" src/zarr`). +- Test: `tests/engine/test_engine_param.py` + +**Interfaces:** +- Consumes: `EngineName`, engines from earlier tasks. +- Produces: public keyword `engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None` on `zarr.create_array`, `zarr.open_array`, `zarr.api.asynchronous.create_array`, `zarr.api.asynchronous.open_array`, threaded to `AsyncArray.__init__` (async instances/names) and `Array._engine_spec` (sync instances/names). Passing a *sync instance* through an async API raises `TypeError`, and vice versa; a *name* is valid everywhere. + +- [ ] **Step 1: Write the failing test** + +```python +# tests/engine/test_engine_param.py +from __future__ import annotations + +import numpy as np +import pytest + +import zarr +from zarr.core.engine import DefaultArrayEngine, DefaultAsyncArrayEngine +from zarr.storage import MemoryStore + + +def test_engine_param_combinations() -> None: + store = MemoryStore() + z = zarr.create_array(store, name="a", shape=(4,), chunks=(2,), dtype="int8", engine="default") + z[:] = np.arange(4, dtype="int8") + assert isinstance(z.engine, DefaultArrayEngine) + + z2 = zarr.open_array(store, path="a", engine="default") + np.testing.assert_array_equal(np.asarray(z2[:]), np.arange(4, dtype="int8")) + assert isinstance(z2.async_array.engine, DefaultAsyncArrayEngine) + + # user-provided sync instance + inst = z2.engine + z3 = zarr.open_array(store, path="a", engine=inst) + assert z3.engine is inst + + +def test_engine_param_unknown_name() -> None: + with pytest.raises(ValueError, match="unknown engine"): + zarr.create_array( + MemoryStore(), shape=(2,), chunks=(2,), dtype="int8", engine="nope" + ) +``` + +- [ ] **Step 2: Implement** + +Thread the keyword: each `create_array`/`open_array` entry point passes +`engine=engine` down to where `AsyncArray(...)`/`Array(...)` is constructed. +Rules: +- Async entry points pass names and `AsyncArrayEngine` instances to + `AsyncArray.__init__`; a sync `ArrayEngine` instance there raises + `TypeError("a sync ArrayEngine cannot serve an AsyncArray; pass a name or an AsyncArrayEngine")`. +- Sync entry points pass names to *both* layers (the `Array` stores the name + for its sync engine; the inner `AsyncArray` also gets the name so async + access stays consistent), and sync instances only to `Array`. +- Detection: `isinstance(engine, str) or engine is None` → both layers; + otherwise check for a coroutine `read_selection` via + `inspect.iscoroutinefunction(engine.read_selection)`. + +- [ ] **Step 3: Run tests** + +Run: `uv run pytest tests/engine/test_engine_param.py -v` — Expected: PASS +Run: `uv run pytest tests/test_api.py -q` — Expected: PASS + +- [ ] **Step 4: Commit** + +```bash +git add src/zarr/core/array.py src/zarr/api tests/engine/test_engine_param.py +git commit -m "feat: engine= parameter on array creation and open APIs" +``` + +--- + +### Task 8: Zarrista engines (`zarr.zarrista`) + +**Files:** +- Create: `src/zarr/zarrista/__init__.py`, `src/zarr/zarrista/_translate.py`, `src/zarr/zarrista/_engine.py` +- Modify: `pyproject.toml` (add `zarrista` dependency group, git-pinned) +- Test: `tests/zarrista/__init__.py` (empty), `tests/zarrista/test_translate.py`, `tests/zarrista/test_engine.py` + +**Interfaces:** +- Consumes: protocols/`Region` (Task 1), `UnsupportedEngineError` (Task 1). +- Produces: `ZarristaHierarchyEngine(store: Store)`, `ZarristaAsyncHierarchyEngine(store: Store)`, `ZarristaEngine`, `ZarristaAsyncEngine` (imported lazily by Task 4's resolver). +- Zarrista API used (from its `.pyi` stubs, main branch): sync + `zarrista.Array.from_metadata(metadata: dict, store, path)`, + `.retrieve_array_subset(selection) -> DecodedArray`, + `.store_chunk(chunk_indices, ArrayBytes)`, `.retrieve_chunk(chunk_indices)`, + `.chunk_subset(chunk_indices) -> tuple[slice, ...]`, `.chunk_grid_shape`, + `.chunk_shape(chunk_indices)`; async twins on `zarrista.AsyncArray` + (`from_metadata`, awaitable retrieve/store). Stores: + `zarrista.FilesystemStore(path)` (sync), obstore `ObjectStore` / icechunk + `Session` (async). `zarrista.ArrayBytes(bytes=...)` wraps decoded chunk bytes. + +- [ ] **Step 1: Add the dependency group** + +In `pyproject.toml`, next to the existing dependency groups (see +`[dependency-groups]`), add (pin the current zarrista main commit hash — get it +with `git ls-remote https://github.com/developmentseed/zarrista HEAD`): + +```toml +zarrista = [ + "zarrista @ git+https://github.com/developmentseed/zarrista@", +] +``` + +Run: `uv sync --group zarrista` — Expected: builds/installs zarrista (Rust +toolchain required; if the build fails locally, note it and let CI cover it — +zarrista publishes no sdist-independent wheels from git). + +- [ ] **Step 2: Write the failing tests** + +```python +# tests/zarrista/test_translate.py +from __future__ import annotations + +import pytest + +zarrista = pytest.importorskip("zarrista") + +from zarr.errors import UnsupportedEngineError +from zarr.storage import LocalStore, MemoryStore +from zarr.zarrista._translate import translate_store_async, translate_store_sync + + +def test_local_store_translates_to_filesystem_store(tmp_path) -> None: + zs = translate_store_sync(LocalStore(tmp_path)) + assert isinstance(zs, zarrista.FilesystemStore) + + +def test_memory_store_rejected_sync(tmp_path) -> None: + with pytest.raises(UnsupportedEngineError, match="MemoryStore"): + translate_store_sync(MemoryStore()) + + +def test_local_store_rejected_async(tmp_path) -> None: + # async side wants obstore/icechunk; LocalStore is sync-only in v1 + with pytest.raises(UnsupportedEngineError): + translate_store_async(LocalStore(tmp_path)) + + +def test_object_store_translates_to_obstore(tmp_path) -> None: + obstore = pytest.importorskip("obstore") + from zarr.storage import ObjectStore + + inner = obstore.store.LocalStore(prefix=str(tmp_path)) + zstore = ObjectStore(inner) + assert translate_store_async(zstore) is inner +``` + +```python +# tests/zarrista/test_engine.py +from __future__ import annotations + +import numpy as np +import pytest + +zarrista = pytest.importorskip("zarrista") + +import zarr +from zarr.errors import UnsupportedEngineError +from zarr.storage import LocalStore + + +def _make(tmp_path, **kwargs): + z = zarr.create_array( + LocalStore(tmp_path), shape=(10, 9), chunks=(3, 4), dtype="float32", **kwargs + ) + z[:, :] = np.arange(90, dtype="float32").reshape(10, 9) + return z + + +def test_zarrista_engine_read_write_combinations(tmp_path) -> None: + z = _make(tmp_path) + ze = zarr.open_array(LocalStore(tmp_path), engine="zarrista") + + # contiguous read + np.testing.assert_array_equal(np.asarray(ze[2:7, 1:5]), np.asarray(z[2:7, 1:5])) + # strided read via bbox + post-index + np.testing.assert_array_equal(np.asarray(ze[1:9:2, ::3]), np.asarray(z[1:9:2, ::3])) + # full-chunk-aligned write + ze[0:3, 0:4] = np.zeros((3, 4), dtype="float32") + np.testing.assert_array_equal(np.asarray(z[0:3, 0:4]), np.zeros((3, 4), dtype="float32")) + # partial-chunk RMW write + ze[1:2, 1:2] = np.float32(99.0) + assert float(z[1, 1]) == 99.0 + assert float(z[0, 0]) == 0.0 # neighbor in same chunk untouched + + +def test_zarrista_rejects_v2(tmp_path) -> None: + zarr.create_array( + LocalStore(tmp_path), shape=(4,), chunks=(2,), dtype="int8", zarr_format=2 + ) + with pytest.raises(UnsupportedEngineError, match="v3"): + zarr.open_array(LocalStore(tmp_path), engine="zarrista") +``` + +Run: `uv run --group zarrista pytest tests/zarrista -v` +Expected: FAIL (`ModuleNotFoundError: zarr.zarrista`) + +- [ ] **Step 3: Implement translation** + +```python +# src/zarr/zarrista/_translate.py +"""Translate zarr-python stores into stores Zarrista can consume.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from zarr.errors import UnsupportedEngineError +from zarr.storage import LocalStore + +if TYPE_CHECKING: + from zarr.abc.store import Store + +_SYNC_SUPPORTED = "LocalStore" +_ASYNC_SUPPORTED = "zarr.storage.ObjectStore (obstore-backed) or an icechunk store" + + +def translate_store_sync(store: Store) -> Any: + """zarr store -> zarrista sync store (`FilesystemStore`).""" + import zarrista + + if isinstance(store, LocalStore): + return zarrista.FilesystemStore(store.root) + raise UnsupportedEngineError( + f"the zarrista sync engine cannot serve a {type(store).__name__}; " + f"supported: {_SYNC_SUPPORTED}. Note: zarr's MemoryStore lives in the " + "Python process and cannot be shared with the Rust extension." + ) + + +def translate_store_async(store: Store) -> Any: + """zarr store -> zarrista async store (obstore `ObjectStore` or icechunk `Session`).""" + from zarr.storage import ObjectStore + + if isinstance(store, ObjectStore): + return store.store # the underlying obstore instance + try: + from icechunk import IcechunkStore # type: ignore[import-not-found] + + if isinstance(store, IcechunkStore): + return store.session + except ImportError: + pass + raise UnsupportedEngineError( + f"the zarrista async engine cannot serve a {type(store).__name__}; " + f"supported: {_ASYNC_SUPPORTED}." + ) +``` + +(Verify the icechunk attribute: `IcechunkStore.session` — check with +`uv run python -c "import icechunk, inspect; print([m for m in dir(icechunk.IcechunkStore) if 'session' in m])"` +if icechunk is installed; otherwise leave the guarded branch as written.) + +- [ ] **Step 4: Implement the engines** + +```python +# src/zarr/zarrista/_engine.py +"""Zarrista-backed array engines.""" + +from __future__ import annotations + +import itertools +from typing import TYPE_CHECKING, Any + +import numpy as np + +from zarr.abc.engine import Region +from zarr.errors import UnsupportedEngineError +from zarr.zarrista._translate import translate_store_async, translate_store_sync + +if TYPE_CHECKING: + from zarr.abc.store import Store + from zarr.core.buffer import BufferPrototype, NDBuffer + from zarr.core.metadata import ArrayMetadata + + +def _require_v3(metadata: ArrayMetadata) -> dict[str, Any]: + if metadata.zarr_format != 3: + raise UnsupportedEngineError( + "the zarrista engine supports Zarr v3 only; this array is " + f"format v{metadata.zarr_format}" + ) + return metadata.to_dict() + + +def _region_to_selection(region: Region) -> tuple[slice, ...]: + return tuple( + slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True) + ) + + +def _decoded_to_numpy(decoded: Any) -> np.ndarray: + # Tensor / (future) VariableArray implement __array__; masked layouts do not + # map to a zarr-python dtype. + type_name = type(decoded).__name__ + if type_name in ("MaskedTensor", "MaskedVariableArray"): + raise NotImplementedError( + f"zarrista returned {type_name}; masked layouts have no " + "zarr-python equivalent" + ) + return np.asarray(decoded) + + +def _chunks_overlapping(region: Region, chunk_shape: tuple[int, ...]) -> Any: + ranges = [ + range(s // c, (e + c - 1) // c) if e > s else range(0) + for s, e, c in zip(region.start, region.end_exclusive, chunk_shape, strict=True) + ] + return itertools.product(*ranges) + + +class ZarristaEngine: + """Sync engine over `zarrista.Array`. No event loop involved.""" + + def __init__(self, zarrista_array: Any) -> None: + self._arr = zarrista_array + + @classmethod + def from_zarr_metadata(cls, store: Store, path: str, metadata: ArrayMetadata) -> ZarristaEngine: + import zarrista + + meta_dict = _require_v3(metadata) + zstore = translate_store_sync(store) + return cls(zarrista.Array.from_metadata(meta_dict, zstore, "/" + path.strip("/"))) + + def with_metadata(self, metadata: ArrayMetadata) -> ZarristaEngine: + import zarrista + + return ZarristaEngine( + zarrista.Array.from_metadata( + _require_v3(metadata), self._arr.store, self._arr.path + ) + ) + + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> Any: + return _decoded_to_numpy( + self._arr.retrieve_array_subset(_region_to_selection(selection)) + ) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + import zarrista + + value_np = np.ascontiguousarray(value.as_ndarray_like()) + # regular grids: the origin chunk has the full (unclipped) chunk shape + chunk_shape = tuple(self._arr.chunk_shape([0] * len(selection.start))) + for chunk_idx in _chunks_overlapping(selection, chunk_shape): + chunk_idx = list(chunk_idx) + chunk_slices = self._arr.chunk_subset(chunk_idx) + # overlap of the write region with this chunk, in array coords + lo = tuple( + max(cs.start, s) + for cs, s in zip(chunk_slices, selection.start, strict=True) + ) + hi = tuple( + min(cs.stop, e) + for cs, e in zip(chunk_slices, selection.end_exclusive, strict=True) + ) + in_value = tuple( + slice(a - s, b - s) + for a, b, s in zip(lo, hi, selection.start, strict=True) + ) + in_chunk = tuple( + slice(a - cs.start, b - cs.start) + for a, b, cs in zip(lo, hi, chunk_slices, strict=True) + ) + full_chunk = all( + sl.start == 0 and sl.stop == (cs.stop - cs.start) + for sl, cs in zip(in_chunk, chunk_slices, strict=True) + ) + if full_chunk: + chunk_np = np.ascontiguousarray(value_np[in_value]) + else: + chunk_np = np.array(_decoded_to_numpy(self._arr.retrieve_chunk(chunk_idx))) + chunk_np[in_chunk] = value_np[in_value] + chunk_np = np.ascontiguousarray(chunk_np) + self._arr.store_chunk(chunk_idx, zarrista.ArrayBytes(chunk_np)) + + +class ZarristaHierarchyEngine: + """Store-bound factory for sync zarrista engines (translates the store once).""" + + def __init__(self, store: Store) -> None: + self._zarr_store = store + self._zstore = translate_store_sync(store) + + def array_engine(self, path: str, metadata: ArrayMetadata) -> ZarristaEngine: + import zarrista + + return ZarristaEngine( + zarrista.Array.from_metadata( + _require_v3(metadata), self._zstore, "/" + path.strip("/") + ) + ) +``` + +`ZarristaAsyncEngine` / `ZarristaAsyncHierarchyEngine`: same shape with +`zarrista.AsyncArray.from_metadata`, `translate_store_async`, and `await` on +`retrieve_array_subset` / `retrieve_chunk` / `store_chunk`. Write the full +async twin classes — do not factor a shared base for v1 (the sync class must +never touch a loop; keeping them textually parallel is clearer). + +Implementation notes: +- `zarrista.ArrayBytes(chunk_np)` — the constructor takes `bytes: Buffer`; + pass the numpy array (it implements the buffer protocol). If zarrista + rejects multi-dimensional buffers, pass + `chunk_np.reshape(-1).view(np.uint8)` — decide by running the test. +- `store_chunk` expects the *full* chunk shape including edge-chunk overhang; + `chunk_subset` returns clipped slices at array edges. For edge chunks the + decoded chunk from `retrieve_chunk` already has the full chunk shape — + patch using `in_chunk` offsets (clipped coords are correct because + `chunk_subset` slices start at the chunk origin). For a *full-chunk* write + on an edge chunk (`full_chunk` computed against the clipped extent), the + written array is smaller than the chunk shape — in that case fall through + to the RMW branch. Adjust `full_chunk` to also require + `(cs.stop - cs.start) == chunk_shape[dim]`. +- `zarrista.Array.from_metadata` path convention is `/`-prefixed (`"/"` is + the root), matching the stub default `path="/"`. +- vlen dtypes: `retrieve_array_subset` returns `VariableArray`; if + `np.asarray` on it fails under the pinned commit, raise + `NotImplementedError("vlen read support pending zarrista numpy export")` + and mark the corresponding differential test `xfail(strict=False)`. + +```python +# src/zarr/zarrista/__init__.py +"""Zarrista-backed array engines for zarr-python. + +Requires the `zarrista` package (`pip install zarr[zarrista]` once released; +currently the git-pinned `zarrista` dependency group). +""" + +from zarr.zarrista._engine import ( + ZarristaAsyncEngine, + ZarristaAsyncHierarchyEngine, + ZarristaEngine, + ZarristaHierarchyEngine, +) + +__all__ = [ + "ZarristaAsyncEngine", + "ZarristaAsyncHierarchyEngine", + "ZarristaEngine", + "ZarristaHierarchyEngine", +] +``` + +- [ ] **Step 5: Run tests** + +Run: `uv run --group zarrista pytest tests/zarrista -v` — Expected: PASS +Run: `uv run pytest tests/zarrista -v` (no group) — Expected: all SKIPPED (importorskip) + +- [ ] **Step 6: Commit** + +```bash +git add src/zarr/zarrista tests/zarrista pyproject.toml uv.lock +git commit -m "feat: zarrista-backed array engines with store translation" +``` + +--- + +### Task 9: Delete `zarr.crud`, `zarr.zarrs`, `packages/zarrs-bindings` + +**Files:** +- Delete: `src/zarr/crud/`, `src/zarr/zarrs/`, `packages/zarrs-bindings/`, `tests/crud/`, `tests/zarrs/`, `.github/workflows/zarrs.yml`, the `changes/` fragments describing zarr.crud/zarr.zarrs (find: `grep -rl "crud\|zarrs" changes/`) +- Modify: `pyproject.toml` — remove the `zarrs` dependency group (line ~138), the `zarrs-bindings` uv source (line ~502), the `/packages/zarrs-bindings` exclude (line ~10), and the `--ignore=src/zarr/zarrs` pytest flag (line ~444). + +- [ ] **Step 1: Delete and unwire** + +```bash +git rm -r src/zarr/crud src/zarr/zarrs packages/zarrs-bindings tests/crud tests/zarrs .github/workflows/zarrs.yml +``` + +Edit `pyproject.toml` as listed above. Search for stragglers: + +```bash +grep -rn "zarr.crud\|zarr\.zarrs\|zarrs_bindings\|zarrs-bindings" src tests docs pyproject.toml .github --include="*" | grep -v superpowers +``` + +Expected: no hits outside `docs/superpowers/` (specs/plans are historical records — leave them). + +- [ ] **Step 2: Full test run** + +Run: `uv sync && uv run pytest tests -x -q --ignore=tests/zarrista` +Expected: PASS (no imports of deleted modules anywhere) +Run: `uv run mypy src/zarr` — Expected: no errors + +- [ ] **Step 3: Commit** + +```bash +git add -u src tests pyproject.toml uv.lock .github +git commit -m "refactor!: remove zarr.crud, zarr.zarrs, and the zarrs-bindings crate" +``` + +--- + +### Task 10: Differential suite, CI, changelog + +**Files:** +- Create: `tests/engine/test_differential.py`, `.github/workflows/engine.yml`, `changes/.feature.md` (get the PR number from the branch's open PR via `gh pr view --json number -q .number`; if no PR exists yet, use the next number after `gh api repos/{owner}/{repo}/issues?state=all&per_page=1 -q '.[0].number'` and note it must match the eventual PR) + +**Interfaces:** +- Consumes: everything. + +- [ ] **Step 1: Differential test** + +```python +# tests/engine/test_differential.py +"""The same operations through both engines must agree with numpy and each other.""" + +from __future__ import annotations + +import numpy as np +import pytest + +import zarr +from zarr.storage import LocalStore + +try: + import zarrista # noqa: F401 + + ENGINES = ["default", "zarrista"] +except ImportError: + ENGINES = ["default"] + +SHAPE = (10, 9) +CHUNKS = (3, 4) + + +@pytest.fixture +def reference(tmp_path): + z = zarr.create_array( + LocalStore(tmp_path), shape=SHAPE, chunks=CHUNKS, dtype="float64" + ) + data = np.arange(90, dtype="float64").reshape(SHAPE) + z[:, :] = data + return tmp_path, data + + +READS = [ + (slice(None), slice(None)), + (slice(2, 7), slice(1, 5)), + (slice(8, 2, -2), slice(None, None, 3)), + (3, slice(None)), + (-1, -2), + (slice(4, 4), slice(None)), +] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("sel", READS) +def test_reads_match_numpy(reference, engine, sel) -> None: + tmp_path, data = reference + z = zarr.open_array(LocalStore(tmp_path), engine=engine) + np.testing.assert_array_equal(np.asarray(z[sel]), data[sel]) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_fancy_reads_match_numpy(reference, engine) -> None: + tmp_path, data = reference + z = zarr.open_array(LocalStore(tmp_path), engine=engine) + np.testing.assert_array_equal( + np.asarray(z.oindex[np.array([7, 1, 4]), np.array([0, 8])]), + data[np.ix_([7, 1, 4], [0, 8])], + ) + np.testing.assert_array_equal( + np.asarray(z.vindex[np.array([9, 0, 3]), np.array([8, 0, 2])]), + data[np.array([9, 0, 3]), np.array([8, 0, 2])], + ) + np.testing.assert_array_equal(np.asarray(z.blocks[1, 2]), data[3:6, 8:9]) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_writes_match_numpy(reference, engine) -> None: + tmp_path, data = reference + z = zarr.open_array(LocalStore(tmp_path), engine=engine) + expected = data.copy() + + z[0:3, 0:4] = 7.0 # aligned full chunk + expected[0:3, 0:4] = 7.0 + z[4:6, 2:9] = np.arange(14, dtype="float64").reshape(2, 7) # partial chunks + expected[4:6, 2:9] = np.arange(14, dtype="float64").reshape(2, 7) + z[1:9:3, ::4] = -1.0 # strided write (facade RMW) + expected[1:9:3, ::4] = -1.0 + + np.testing.assert_array_equal(np.asarray(z[:, :]), expected) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_sharded_reads(reference, engine, tmp_path_factory) -> None: + path = tmp_path_factory.mktemp("sharded") + z = zarr.create_array( + LocalStore(path), + shape=SHAPE, + chunks=(3, 4), # inner chunks + shards=(6, 8), # shard shape + dtype="int32", + ) + data = np.arange(90, dtype="int32").reshape(SHAPE) + z[:, :] = data + zr = zarr.open_array(LocalStore(path), engine=engine) + np.testing.assert_array_equal(np.asarray(zr[2:8, 3:9]), data[2:8, 3:9]) +``` + +Run: `uv run --group zarrista pytest tests/engine/test_differential.py -v` — Expected: PASS (both engines) +Run: `uv run pytest tests/engine/test_differential.py -v` — Expected: PASS (default only) + +- [ ] **Step 2: CI workflow** + +Create `.github/workflows/engine.yml` modeled on the deleted `zarrs.yml` +(check its triggers/paths in git history: `git show HEAD~1:.github/workflows/zarrs.yml`): +a single job on `ubuntu-latest` that installs Rust (`dtolnay/rust-toolchain@stable`), +sets up uv (`astral-sh/setup-uv`), runs +`uv sync --group zarrista` and +`uv run --group zarrista pytest tests/engine tests/zarrista -v`. +Trigger on pull requests touching `src/zarr/**`, `tests/engine/**`, +`tests/zarrista/**`, `pyproject.toml`, and the workflow file itself. Keep +`permissions: {}` at top level and pin action SHAs, matching the repo's other +workflows (zizmor enforces this). + +- [ ] **Step 3: Changelog fragment** + +```markdown +# changes/.feature.md +`Array` and `AsyncArray` now route data I/O through pluggable *array engines* +(`zarr.abc.engine.ArrayEngine` / `AsyncArrayEngine`). The default engine +preserves existing behavior on every store and format. Pass +`engine="zarrista"` (requires the `zarrista` package) to serve Zarr v3 arrays +on local-filesystem, obstore, or icechunk storage through the Rust `zarrs` +implementation. The experimental `zarr.crud` and `zarr.zarrs` modules and the +bundled `zarrs-bindings` crate are removed in favor of this interface. +``` + +- [ ] **Step 4: Full local gate** + +Run: `uv run --group zarrista pytest tests -q` — Expected: PASS +Run: `uv run mypy src/zarr` — Expected: no errors +Run: `uv run pre-commit run --all-files` — Expected: PASS + +- [ ] **Step 5: Commit** + +```bash +git add tests/engine/test_differential.py .github/workflows/engine.yml changes/ +git commit -m "test/ci: engine differential suite and zarrista CI job" +``` + +--- + +## Self-Review Notes (already applied) + +- Spec coverage: protocols+Region (T1), normalization/facade rules (T2, T5, T6), + default engines incl. sync adapter (T3), explicit engine selection without a + config key (T4, T7), truly-sync path (T6), zarrista store translation / + v3-only / RMW writes / buffer handling (T8), fail-loud errors (T1, T4, T8), + deletions (T9), differential tests + CI + changelog (T10). `with_metadata` + rebinding covered in T3/T5/T6. +- Deliberately out of scope, per spec non-goals: group operations, engine-owned + metadata mutation, `out=` in the engine contract (facade copies instead). +- Known judgment calls the implementer may hit: exact `Array` constructor shape + (T6 step 1), zarrista `ArrayBytes` buffer dimensionality (T8 note), icechunk + session attribute (T8 note), vlen export under the pinned commit (T8 note). + Each has a decision rule written at the point of use. From 623c7797c6abd186f781eb48445cf63e26de9d99 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 14:29:39 +0200 Subject: [PATCH 47/75] feat: Region interchange and array/hierarchy engine protocols Add Region class, ArrayEngine, AsyncArrayEngine, HierarchyEngine, and AsyncHierarchyEngine protocols to enable pluggable array engines in zarr-python. Also add UnsupportedEngineError exception for engine construction failures. Assisted-by: ClaudeCode:claude-haiku-4-5 --- src/zarr/abc/engine.py | 91 ++++++++++++++++++++++++++++++++++ src/zarr/errors.py | 9 ++++ tests/engine/__init__.py | 0 tests/engine/test_protocols.py | 48 ++++++++++++++++++ 4 files changed, 148 insertions(+) create mode 100644 src/zarr/abc/engine.py create mode 100644 tests/engine/__init__.py create mode 100644 tests/engine/test_protocols.py diff --git a/src/zarr/abc/engine.py b/src/zarr/abc/engine.py new file mode 100644 index 0000000000..14341f431f --- /dev/null +++ b/src/zarr/abc/engine.py @@ -0,0 +1,91 @@ +"""Array engine protocols. + +An *array engine* owns the data path of one open array: reading and writing +decoded data for contiguous regions. `Array` wraps an object satisfying +`ArrayEngine`; `AsyncArray` wraps an object satisfying `AsyncArrayEngine`. +A *hierarchy engine* is bound to a store and mints array engines that share +resources. See `docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md`. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, NamedTuple, Protocol, runtime_checkable + +if TYPE_CHECKING: + from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer + from zarr.core.metadata import ArrayMetadata + +__all__ = [ + "ArrayEngine", + "AsyncArrayEngine", + "AsyncHierarchyEngine", + "HierarchyEngine", + "Region", +] + + +class Region(NamedTuple): + """A contiguous, step-1 box in array-element coordinates. + + One entry per dimension; `start` is inclusive, `end_exclusive` exclusive. + Callers pass normalized values: non-negative and clipped to the array + shape. This is the only selection type that crosses the engine boundary. + """ + + start: tuple[int, ...] + end_exclusive: tuple[int, ...] + + @property + def shape(self) -> tuple[int, ...]: + """The ndim-preserving shape of the box.""" + return tuple(e - s for s, e in zip(self.start, self.end_exclusive, strict=True)) + + +@runtime_checkable +class ArrayEngine(Protocol): + """The synchronous data path of one open array. + + Bound to `(store, path, metadata)` at construction. Methods must not + require a running event loop. Read results are ndim-preserving with + shape `selection.shape` and need only implement `__array__`. + + Note: `runtime_checkable` isinstance checks only verify method names; + mypy is the authoritative conformance check. + """ + + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: ... + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: ... + + def with_metadata(self, metadata: ArrayMetadata) -> ArrayEngine: ... + + +@runtime_checkable +class AsyncArrayEngine(Protocol): + """The asynchronous data path of one open array. See `ArrayEngine`.""" + + async def read_selection( + self, selection: Region, *, prototype: BufferPrototype + ) -> NDArrayLike: ... + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: ... + + def with_metadata(self, metadata: ArrayMetadata) -> AsyncArrayEngine: ... + + +@runtime_checkable +class HierarchyEngine(Protocol): + """A store-bound factory for synchronous array engines.""" + + def array_engine(self, path: str, metadata: ArrayMetadata) -> ArrayEngine: ... + + +@runtime_checkable +class AsyncHierarchyEngine(Protocol): + """A store-bound factory for asynchronous array engines.""" + + def array_engine(self, path: str, metadata: ArrayMetadata) -> AsyncArrayEngine: ... diff --git a/src/zarr/errors.py b/src/zarr/errors.py index 781bebe534..76f7807b88 100644 --- a/src/zarr/errors.py +++ b/src/zarr/errors.py @@ -13,6 +13,7 @@ "NegativeStepError", "NodeTypeValidationError", "UnstableSpecificationWarning", + "UnsupportedEngineError", "VindexInvalidSelectionError", "ZarrDeprecationWarning", "ZarrFutureWarning", @@ -155,3 +156,11 @@ class ChunkNotFoundError(BaseZarrError): """ Raised when a chunk that was expected to exist in storage was not retrieved successfully. """ + + +class UnsupportedEngineError(ValueError): + """Raised when an array engine cannot serve the requested store or array. + + Examples: a store the engine cannot translate, or metadata (e.g. Zarr v2) + the engine does not support. Raised at engine construction time. + """ diff --git a/tests/engine/__init__.py b/tests/engine/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/engine/test_protocols.py b/tests/engine/test_protocols.py new file mode 100644 index 0000000000..d8badc3a04 --- /dev/null +++ b/tests/engine/test_protocols.py @@ -0,0 +1,48 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +from zarr.abc.engine import ArrayEngine, AsyncArrayEngine, Region +from zarr.errors import UnsupportedEngineError + +if TYPE_CHECKING: + from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer + from zarr.core.metadata import ArrayMetadata + + +def test_region() -> None: + """Region carries start/end_exclusive and derives shape.""" + r = Region(start=(1, 2), end_exclusive=(4, 2)) + assert r.start == (1, 2) + assert r.end_exclusive == (4, 2) + assert r.shape == (3, 0) + + +class _FakeSyncEngine: + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + return np.zeros(selection.shape) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + return None + + def with_metadata(self, metadata: ArrayMetadata) -> _FakeSyncEngine: + return self + + +def test_runtime_checkable_protocols() -> None: + """isinstance checks verify method presence for the sync protocol.""" + assert isinstance(_FakeSyncEngine(), ArrayEngine) + assert not isinstance(object(), ArrayEngine) + assert ( + not isinstance(_FakeSyncEngine(), AsyncArrayEngine) or True + ) # names match; mypy is authoritative + + +def test_unsupported_engine_error_is_value_error() -> None: + with pytest.raises(ValueError): + raise UnsupportedEngineError("nope") From 5aeb8005b9a82be7ad8fae218ae22d14eb0d88c2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 14:44:42 +0200 Subject: [PATCH 48/75] feat: normalize selection kinds to Region + post-index Adds `zarr.core.engine` with helpers that reduce basic, orthogonal, coordinate, and block selections to a `(Region, post_index)` pair, the shape Task 5/6 engine reads and writes will consume. `normalize_basic` adapts the box math from `zarr.crud._api._normalize_selection` (kept until Task 9 removes that package). `normalize_orthogonal` widens integer axes to length-1 ranges and records them via a `_Squeeze` marker so `apply_post_index` reproduces numpy's oindex axis-dropping; `strip_squeeze` peels that marker off for write-side callers that index a target array directly instead of going through `apply_post_index`. Assisted-by: ClaudeCode:claude-sonnet-5 --- src/zarr/core/engine/__init__.py | 17 +++ src/zarr/core/engine/_normalize.py | 206 +++++++++++++++++++++++++++++ tests/engine/test_normalize.py | 117 ++++++++++++++++ 3 files changed, 340 insertions(+) create mode 100644 src/zarr/core/engine/__init__.py create mode 100644 src/zarr/core/engine/_normalize.py create mode 100644 tests/engine/test_normalize.py diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py new file mode 100644 index 0000000000..be7274bdc1 --- /dev/null +++ b/src/zarr/core/engine/__init__.py @@ -0,0 +1,17 @@ +from zarr.core.engine._normalize import ( + apply_post_index, + normalize_basic, + normalize_block, + normalize_coordinate, + normalize_orthogonal, + strip_squeeze, +) + +__all__ = [ + "apply_post_index", + "normalize_basic", + "normalize_block", + "normalize_coordinate", + "normalize_orthogonal", + "strip_squeeze", +] diff --git a/src/zarr/core/engine/_normalize.py b/src/zarr/core/engine/_normalize.py new file mode 100644 index 0000000000..612a2ecadb --- /dev/null +++ b/src/zarr/core/engine/_normalize.py @@ -0,0 +1,206 @@ +"""Reduce public selection kinds to `(Region, post_index)` pairs. + +The engine boundary only speaks contiguous step-1 boxes (`Region`). Each +helper returns the box to transfer plus the numpy index that, applied to the +ndim-preserving box result, yields exactly `array[original_selection]`. +""" + +from __future__ import annotations + +import operator +import types +from typing import TYPE_CHECKING, Any + +import numpy as np + +from zarr.abc.engine import Region + +if TYPE_CHECKING: + from zarr.core.indexing import BasicSelection + + +def _expand(selection: Any, ndim: int) -> tuple[Any, ...]: + """Expand Ellipsis and pad missing trailing axes with full slices.""" + sel_tuple = selection if isinstance(selection, tuple) else (selection,) + n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) + if n_ellipsis > 1: + raise IndexError("an index can only have a single ellipsis ('...')") + if n_ellipsis == 1: + i = sel_tuple.index(Ellipsis) + n_fill = ndim - (len(sel_tuple) - 1) + if n_fill < 0: + raise IndexError(f"too many indices for array: array is {ndim}-dimensional") + sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] + if len(sel_tuple) > ndim: + raise IndexError(f"too many indices for array: array is {ndim}-dimensional") + return sel_tuple + (slice(None),) * (ndim - len(sel_tuple)) + + +def _normalize_int(sel: Any, size: int, dim: int) -> int: + idx = operator.index(sel) + if idx < 0: + idx += size + if not 0 <= idx < size: + raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") + return idx + + +def normalize_basic( + selection: BasicSelection, shape: tuple[int, ...] +) -> tuple[Region, tuple[slice | int, ...]]: + """Normalize a numpy basic-indexing selection to a step-1 box. + + Supports integers, slices (any step), and `Ellipsis`. Integer axes get a + length-1 range in the box and `0` in the post index (dropping the axis, + matching numpy). Fancy elements raise `TypeError`. + """ + sel_tuple = _expand(selection, len(shape)) + starts: list[int] = [] + ends: list[int] = [] + post: list[slice | int] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + start, stop, step = sel.indices(size) + n = len(range(start, stop, step)) + if n == 0: + starts.append(0) + ends.append(0) + post.append(slice(None)) + elif step > 0: + starts.append(start) + ends.append(start + (n - 1) * step + 1) + post.append(slice(None, None, step)) + else: + last = start + (n - 1) * step + starts.append(last) + ends.append(start + 1) + post.append(slice(None, None, step)) + elif isinstance(sel, types.EllipsisType): + raise AssertionError("Ellipsis expanded by _expand") # noqa: TRY004 + else: + try: + idx = _normalize_int(sel, size, dim) + except TypeError: + raise TypeError( + f"unsupported selection element {sel!r}: only integers, " + "slices, and Ellipsis are supported by basic indexing" + ) from None + starts.append(idx) + ends.append(idx + 1) + post.append(0) + return Region(start=tuple(starts), end_exclusive=tuple(ends)), tuple(post) + + +class _Squeeze: + """Marker appended to an orthogonal post index: squeeze these axes.""" + + def __init__(self, axes: tuple[int, ...]) -> None: + self.axes = axes + + +def normalize_orthogonal(selection: Any, shape: tuple[int, ...]) -> tuple[Region, tuple[Any, ...]]: + """Normalize an orthogonal (`oindex`) selection to a box + outer index. + + Each axis selector may be an integer, a slice, an integer array, or a 1-d + boolean mask for that axis. The post index is the `np.ix_`-broadcastable + tuple of per-axis integer arrays (relative to the box origin); if any axis + was an integer, a trailing `_Squeeze` marker records which axes to drop + afterwards. Use `apply_post_index` to apply the result correctly. + """ + sel_tuple = _expand(selection, len(shape)) + starts: list[int] = [] + ends: list[int] = [] + axis_indices: list[np.ndarray] = [] + for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): + if isinstance(sel, slice): + idxs = np.arange(*sel.indices(size)) + elif isinstance(sel, np.ndarray) and sel.dtype == bool: + if sel.ndim != 1 or sel.shape[0] != size: + raise IndexError(f"boolean index for axis {dim} must be 1-d with length {size}") + idxs = np.nonzero(sel)[0] + elif isinstance(sel, (np.ndarray, list)): + idxs = np.asarray(sel, dtype=np.intp) + if idxs.ndim != 1: + raise IndexError(f"orthogonal index for axis {dim} must be 1-d") + idxs = np.where(idxs < 0, idxs + size, idxs) + if idxs.size and (idxs.min() < 0 or idxs.max() >= size): + raise IndexError(f"index out of bounds for axis {dim} with size {size}") + else: + idxs = np.array([_normalize_int(sel, size, dim)], dtype=np.intp) + if idxs.size == 0: + starts.append(0) + ends.append(0) + axis_indices.append(idxs) + else: + lo, hi = int(idxs.min()), int(idxs.max()) + starts.append(lo) + ends.append(hi + 1) + axis_indices.append(idxs - lo) + region = Region(start=tuple(starts), end_exclusive=tuple(ends)) + post = np.ix_(*axis_indices) if axis_indices else () + # integer axes were widened to length-1 arrays; the caller squeezes them + squeeze_axes = tuple( + i for i, sel in enumerate(sel_tuple) if not isinstance(sel, (slice, np.ndarray, list)) + ) + if squeeze_axes: + return region, (*post, _Squeeze(squeeze_axes)) + return region, post + + +def apply_post_index(box: np.ndarray, post: tuple[Any, ...]) -> np.ndarray: + """Apply a post index produced by a `normalize_*` helper to a box read.""" + if post and isinstance(post[-1], _Squeeze): + result = box[post[:-1]] if len(post) > 1 else box + return np.squeeze(result, axis=post[-1].axes) + if post == (): + return box + return np.asarray(box[post]) + + +def strip_squeeze(post: tuple[Any, ...]) -> tuple[Any, ...]: + """Return `post` without a trailing `_Squeeze` marker, if present. + + Writes apply the post index directly to a target array (not through + `apply_post_index`), so callers that only need the "real" numpy index + strip the `_Squeeze` marker first. Returns `post` unchanged when there is + no marker. + """ + if post and isinstance(post[-1], _Squeeze): + return post[:-1] + return post + + +def normalize_coordinate( + selection: tuple[Any, ...], shape: tuple[int, ...] +) -> tuple[Region, tuple[np.ndarray, ...]]: + """Normalize a coordinate (`vindex`) selection to a box + pointwise index.""" + coords = tuple(np.asarray(c, dtype=np.intp) for c in selection) + if len(coords) != len(shape): + raise IndexError(f"coordinate selection needs {len(shape)} axis arrays, got {len(coords)}") + coords = tuple(np.where(c < 0, c + size, c) for c, size in zip(coords, shape, strict=True)) + for dim, (c, size) in enumerate(zip(coords, shape, strict=True)): + if c.size and (c.min() < 0 or c.max() >= size): + raise IndexError(f"index out of bounds for axis {dim} with size {size}") + starts = tuple(int(c.min()) if c.size else 0 for c in coords) + ends = tuple(int(c.max()) + 1 if c.size else 0 for c in coords) + post = tuple(c - s for c, s in zip(coords, starts, strict=True)) + return Region(start=starts, end_exclusive=ends), post + + +def normalize_block( + block_coords: tuple[int, ...], + *, + chunk_grid_shape: tuple[int, ...], + chunk_shape: tuple[int, ...], + shape: tuple[int, ...], +) -> Region: + """Normalize a block (chunk-grid) selection to its contiguous box.""" + starts = [] + ends = [] + for dim, (b, nblocks, csize, size) in enumerate( + zip(block_coords, chunk_grid_shape, chunk_shape, shape, strict=True) + ): + idx = _normalize_int(b, nblocks, dim) + starts.append(idx * csize) + ends.append(min((idx + 1) * csize, size)) + return Region(start=tuple(starts), end_exclusive=tuple(ends)) diff --git a/tests/engine/test_normalize.py b/tests/engine/test_normalize.py new file mode 100644 index 0000000000..d562dd0105 --- /dev/null +++ b/tests/engine/test_normalize.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np +import numpy.typing as npt +import pytest + +from zarr.core.engine import ( + apply_post_index, + normalize_basic, + normalize_block, + normalize_coordinate, + normalize_orthogonal, + strip_squeeze, +) +from zarr.core.engine._normalize import _Squeeze + +if TYPE_CHECKING: + from zarr.abc.engine import Region + +SHAPE = (10, 9) +ARR = np.arange(90).reshape(SHAPE) + + +def _read_box(region: Region) -> npt.NDArray[np.int_]: + """Simulate an engine read: the ndim-preserving box.""" + return ARR[tuple(slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True))] + + +@pytest.mark.parametrize( + "sel", + [ + (slice(2, 7), slice(0, 9)), + (slice(None), slice(3, 4)), + (slice(8, 2, -2), slice(None)), + (slice(1, 8, 3), slice(2, 9, 2)), + (3, slice(None)), + (-1, -2), + (Ellipsis, 4), + slice(5), + Ellipsis, + (slice(4, 4), slice(None)), # empty + ], +) +def test_normalize_basic_matches_numpy(sel: Any) -> None: + region, post = normalize_basic(sel, SHAPE) + np.testing.assert_array_equal(apply_post_index(_read_box(region), post), ARR[sel]) + + +@pytest.mark.parametrize( + "sel", + [ + (np.array([1, 4, 7]), np.array([0, 8])), + (np.array([7, 1, 4]), slice(2, 6)), # unordered axis indices + (np.array([3, 3]), np.array([5, 5])), # repeats + (slice(1, 9, 2), np.array([2])), + (np.array([0]), 4), # int axis + ], +) +def test_normalize_orthogonal_matches_numpy_oindex(sel: Any) -> None: + from zarr.core.indexing import oindex as _reference_oindex + + region, post = normalize_orthogonal(sel, SHAPE) + # `zarr.core.indexing.oindex` is the existing, well-tested reference + # implementation of orthogonal indexing (np.ix_ outer indexing with + # integer axes dropped afterwards) -- use it as the ground truth rather + # than reimplementing it by hand. + expected = _reference_oindex(ARR, sel if isinstance(sel, tuple) else (sel,)) + np.testing.assert_array_equal(apply_post_index(_read_box(region), post), expected) + + +def test_normalize_coordinate_matches_numpy_vindex() -> None: + coords = (np.array([9, 0, 3, 3]), np.array([8, 0, 2, 2])) + region, post = normalize_coordinate(coords, SHAPE) + np.testing.assert_array_equal(apply_post_index(_read_box(region), post), ARR[coords]) + + +def test_normalize_block_is_contiguous() -> None: + # chunk shape (3, 4) over SHAPE (10, 9): block (1, 2) spans rows 3:6, cols 8:9 + region = normalize_block((1, 2), chunk_grid_shape=(4, 3), chunk_shape=(3, 4), shape=SHAPE) + assert region.start == (3, 8) + assert region.end_exclusive == (6, 9) + + +def test_normalize_basic_rejects_fancy() -> None: + with pytest.raises(TypeError): + normalize_basic((np.array([1, 2]), slice(None)), SHAPE) # type: ignore[arg-type] + + +def test_normalize_basic_rejects_out_of_bounds_int() -> None: + with pytest.raises(IndexError): + normalize_basic((10, slice(None)), SHAPE) + + +def test_normalize_coordinate_rejects_out_of_bounds() -> None: + with pytest.raises(IndexError): + normalize_coordinate((np.array([10]), np.array([0])), SHAPE) + + +def test_normalize_orthogonal_rejects_boolean_ndim_mismatch() -> None: + with pytest.raises(IndexError): + normalize_orthogonal((np.zeros((2, 2), dtype=bool), slice(None)), SHAPE) + + +def test_strip_squeeze_removes_trailing_marker() -> None: + post_with_marker = (slice(None), np.array([0]), _Squeeze((1,))) + post_without = (slice(None), np.array([0])) + stripped = strip_squeeze(post_with_marker) + assert len(stripped) == len(post_without) + assert stripped[0] == post_without[0] + np.testing.assert_array_equal(stripped[1], post_without[1]) + + +def test_strip_squeeze_identity_when_no_marker() -> None: + post = (slice(None), slice(2, 4)) + assert strip_squeeze(post) == post From 5c7147a0be8a4af1bb88796d0645ccbc52d12b4c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 14:52:33 +0200 Subject: [PATCH 49/75] fix: handle list of booleans in normalize_orthogonal Fix silent misbehavior where plain Python lists of booleans were cast to integer indices (0/1) instead of treated as boolean masks. Refactored the array-like branch to check the converted array's dtype and branch on bool vs integer-castable types. Added two test cases: one verifying list-of-bools produces the same result as equivalent np.ndarray bool mask, one ensuring wrong-length bool list raises IndexError. Assisted-by: ClaudeCode:claude-haiku-4-5 --- src/zarr/core/engine/_normalize.py | 25 ++++++++++++++---------- tests/engine/test_normalize.py | 31 ++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 10 deletions(-) diff --git a/src/zarr/core/engine/_normalize.py b/src/zarr/core/engine/_normalize.py index 612a2ecadb..56273f1f1c 100644 --- a/src/zarr/core/engine/_normalize.py +++ b/src/zarr/core/engine/_normalize.py @@ -114,17 +114,22 @@ def normalize_orthogonal(selection: Any, shape: tuple[int, ...]) -> tuple[Region for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): if isinstance(sel, slice): idxs = np.arange(*sel.indices(size)) - elif isinstance(sel, np.ndarray) and sel.dtype == bool: - if sel.ndim != 1 or sel.shape[0] != size: - raise IndexError(f"boolean index for axis {dim} must be 1-d with length {size}") - idxs = np.nonzero(sel)[0] elif isinstance(sel, (np.ndarray, list)): - idxs = np.asarray(sel, dtype=np.intp) - if idxs.ndim != 1: - raise IndexError(f"orthogonal index for axis {dim} must be 1-d") - idxs = np.where(idxs < 0, idxs + size, idxs) - if idxs.size and (idxs.min() < 0 or idxs.max() >= size): - raise IndexError(f"index out of bounds for axis {dim} with size {size}") + # Convert to array first to check dtype (handles list of bools) + sel_array = np.asarray(sel) + if sel_array.dtype == bool: + # Boolean mask path + if sel_array.ndim != 1 or sel_array.shape[0] != size: + raise IndexError(f"boolean index for axis {dim} must be 1-d with length {size}") + idxs = np.nonzero(sel_array)[0] + else: + # Integer index path + idxs = np.asarray(sel, dtype=np.intp) + if idxs.ndim != 1: + raise IndexError(f"orthogonal index for axis {dim} must be 1-d") + idxs = np.where(idxs < 0, idxs + size, idxs) + if idxs.size and (idxs.min() < 0 or idxs.max() >= size): + raise IndexError(f"index out of bounds for axis {dim} with size {size}") else: idxs = np.array([_normalize_int(sel, size, dim)], dtype=np.intp) if idxs.size == 0: diff --git a/tests/engine/test_normalize.py b/tests/engine/test_normalize.py index d562dd0105..90411f2b42 100644 --- a/tests/engine/test_normalize.py +++ b/tests/engine/test_normalize.py @@ -115,3 +115,34 @@ def test_strip_squeeze_removes_trailing_marker() -> None: def test_strip_squeeze_identity_when_no_marker() -> None: post = (slice(None), slice(2, 4)) assert strip_squeeze(post) == post + + +def test_normalize_orthogonal_list_of_bools_matches_array_bool_mask() -> None: + """List of booleans should produce the same result as np.ndarray bool mask.""" + from zarr.core.indexing import oindex as _reference_oindex + + # Test case from the defect report: list of 10 booleans for axis 0 + bool_list = [True, False, True] + [False] * 7 + bool_array = np.array(bool_list, dtype=bool) + + # Both should produce the same result when used as selectors + region_list, post_list = normalize_orthogonal((bool_list, slice(None)), SHAPE) + region_array, post_array = normalize_orthogonal((bool_array, slice(None)), SHAPE) + + # Reference implementation using zarr.core.indexing.oindex + expected = _reference_oindex(ARR, (bool_array, slice(None))) + + # List of bools should match the array bool mask result + result_list = apply_post_index(_read_box(region_list), post_list) + result_array = apply_post_index(_read_box(region_array), post_array) + + np.testing.assert_array_equal(result_list, expected) + np.testing.assert_array_equal(result_array, expected) + + +def test_normalize_orthogonal_wrong_length_list_of_bools_raises_error() -> None: + """Wrong-length list of booleans should raise IndexError like ndarray.""" + wrong_length_bool_list = [True, False, True] # Length 3, but axis 0 has size 10 + + with pytest.raises(IndexError, match="boolean index for axis 0 must be 1-d with length 10"): + normalize_orthogonal((wrong_length_bool_list, slice(None)), SHAPE) From aeb562e78dbd9ce76772d76b09092cd39786e3bd Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 15:01:48 +0200 Subject: [PATCH 50/75] feat: default array engines over the codec pipeline Add DefaultAsyncArrayEngine/DefaultArrayEngine and their hierarchy-engine factories in zarr.core.engine, wrapping the existing codec-pipeline read/write machinery behind the ArrayEngine/AsyncArrayEngine protocols introduced in earlier tasks of the array-engine-protocol work. Assisted-by: ClaudeCode:claude-sonnet-5 --- src/zarr/core/engine/__init__.py | 10 ++ src/zarr/core/engine/_default.py | 176 ++++++++++++++++++++++++++++ tests/engine/test_default_engine.py | 64 ++++++++++ 3 files changed, 250 insertions(+) create mode 100644 src/zarr/core/engine/_default.py create mode 100644 tests/engine/test_default_engine.py diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py index be7274bdc1..ed16e4bbad 100644 --- a/src/zarr/core/engine/__init__.py +++ b/src/zarr/core/engine/__init__.py @@ -1,3 +1,9 @@ +from zarr.core.engine._default import ( + DefaultArrayEngine, + DefaultAsyncArrayEngine, + DefaultAsyncHierarchyEngine, + DefaultHierarchyEngine, +) from zarr.core.engine._normalize import ( apply_post_index, normalize_basic, @@ -8,6 +14,10 @@ ) __all__ = [ + "DefaultArrayEngine", + "DefaultAsyncArrayEngine", + "DefaultAsyncHierarchyEngine", + "DefaultHierarchyEngine", "apply_post_index", "normalize_basic", "normalize_block", diff --git a/src/zarr/core/engine/_default.py b/src/zarr/core/engine/_default.py new file mode 100644 index 0000000000..117f6da19c --- /dev/null +++ b/src/zarr/core/engine/_default.py @@ -0,0 +1,176 @@ +"""The default engines: today's codec-pipeline machinery behind the protocol.""" + +from __future__ import annotations + +from dataclasses import replace +from typing import TYPE_CHECKING + +from zarr.core.array import _get_chunk_spec, create_codec_pipeline +from zarr.core.array_spec import ArraySpec, parse_array_config +from zarr.core.chunk_grids import ChunkGrid +from zarr.core.common import product +from zarr.core.indexing import BasicIndexer +from zarr.core.sync import sync +from zarr.storage._common import StorePath + +if TYPE_CHECKING: + from zarr.abc.engine import Region + from zarr.abc.store import Store + from zarr.core.array_spec import ArrayConfig + from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer + from zarr.core.metadata import ArrayMetadata + +__all__ = [ + "DefaultArrayEngine", + "DefaultAsyncArrayEngine", + "DefaultAsyncHierarchyEngine", + "DefaultHierarchyEngine", +] + + +def _region_to_slices(region: Region) -> tuple[slice, ...]: + return tuple(slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True)) + + +class DefaultAsyncArrayEngine: + """Codec-pipeline-backed engine. Any store, Zarr v2 and v3.""" + + def __init__(self, store_path: StorePath, metadata: ArrayMetadata, config: ArrayConfig) -> None: + self.store_path = store_path + self.metadata = metadata + self.config = config + self._chunk_grid = ChunkGrid.from_metadata(metadata) + self._codec_pipeline = create_codec_pipeline(metadata=metadata, store=store_path.store) + + def with_metadata(self, metadata: ArrayMetadata) -> DefaultAsyncArrayEngine: + return DefaultAsyncArrayEngine( + store_path=self.store_path, metadata=metadata, config=self.config + ) + + def _indexer(self, region: Region) -> BasicIndexer: + return BasicIndexer( + _region_to_slices(region), + shape=self.metadata.shape, + chunk_grid=self._chunk_grid, + ) + + def _regular_chunk_spec( + self, config: ArrayConfig, prototype: BufferPrototype + ) -> ArraySpec | None: + """Same optimization as `_get_selection`/`_set_selection`: build one shared + `ArraySpec` for regular chunk grids instead of a per-chunk lookup. + """ + if not self._chunk_grid.is_regular: + return None + return ArraySpec( + shape=self._chunk_grid.chunk_shape, + dtype=self.metadata.dtype, + fill_value=self.metadata.fill_value, + config=config, + prototype=prototype, + ) + + def _v2_order_config(self) -> ArrayConfig: + # need to use the order from the metadata for v2 (mirrors _get_selection/_set_selection) + if self.metadata.zarr_format == 2: + return replace(self.config, order=self.metadata.order) + return self.config + + async def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + indexer = self._indexer(selection) + if self.metadata.zarr_format == 2: + dtype = self.metadata.dtype.to_native_dtype() + order = self.metadata.order + else: + dtype = self.metadata.data_type.to_native_dtype() + order = self.config.order + out_buffer = prototype.nd_buffer.empty(shape=indexer.shape, dtype=dtype, order=order) + if product(indexer.shape) > 0: + _config = self._v2_order_config() + regular_chunk_spec = self._regular_chunk_spec(_config, prototype) + await self._codec_pipeline.read( + [ + ( + self.store_path / self.metadata.encode_chunk_key(chunk_coords), + regular_chunk_spec + if regular_chunk_spec is not None + else _get_chunk_spec( + self.metadata, self._chunk_grid, chunk_coords, _config, prototype + ), + chunk_selection, + out_selection, + is_complete_chunk, + ) + for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexer + ], + out_buffer, + drop_axes=indexer.drop_axes, + ) + return out_buffer.as_ndarray_like() + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + indexer = self._indexer(selection) + _config = self._v2_order_config() + regular_chunk_spec = self._regular_chunk_spec(_config, prototype) + await self._codec_pipeline.write( + [ + ( + self.store_path / self.metadata.encode_chunk_key(chunk_coords), + regular_chunk_spec + if regular_chunk_spec is not None + else _get_chunk_spec( + self.metadata, self._chunk_grid, chunk_coords, _config, prototype + ), + chunk_selection, + out_selection, + is_complete_chunk, + ) + for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexer + ], + value, + drop_axes=indexer.drop_axes, + ) + + +class DefaultArrayEngine: + """Sync adapter over `DefaultAsyncArrayEngine` via `sync()`.""" + + def __init__(self, async_engine: DefaultAsyncArrayEngine) -> None: + self._async = async_engine + + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + return sync(self._async.read_selection(selection, prototype=prototype)) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + sync(self._async.write_selection(selection, value, prototype=prototype)) + + def with_metadata(self, metadata: ArrayMetadata) -> DefaultArrayEngine: + return DefaultArrayEngine(self._async.with_metadata(metadata)) + + +class DefaultAsyncHierarchyEngine: + """Store-bound factory for default async engines.""" + + def __init__(self, store: Store) -> None: + self.store = store + + def array_engine(self, path: str, metadata: ArrayMetadata) -> DefaultAsyncArrayEngine: + return DefaultAsyncArrayEngine( + store_path=StorePath(self.store, path), + metadata=metadata, + config=parse_array_config(None), + ) + + +class DefaultHierarchyEngine: + """Store-bound factory for default sync engines.""" + + def __init__(self, store: Store) -> None: + self._async = DefaultAsyncHierarchyEngine(store) + + def array_engine(self, path: str, metadata: ArrayMetadata) -> DefaultArrayEngine: + return DefaultArrayEngine(self._async.array_engine(path, metadata)) diff --git a/tests/engine/test_default_engine.py b/tests/engine/test_default_engine.py new file mode 100644 index 0000000000..26f2c2a8be --- /dev/null +++ b/tests/engine/test_default_engine.py @@ -0,0 +1,64 @@ +from __future__ import annotations + +from typing import Any + +import numpy as np + +import zarr +from zarr.abc.engine import AsyncArrayEngine, Region +from zarr.core.buffer import default_buffer_prototype +from zarr.core.engine import DefaultArrayEngine, DefaultAsyncArrayEngine +from zarr.core.sync import sync +from zarr.storage import MemoryStore + + +def _make_array() -> zarr.Array[Any]: + z = zarr.create_array(MemoryStore(), shape=(10, 9), chunks=(3, 4), dtype="int32", fill_value=0) + z[:, :] = np.arange(90, dtype="int32").reshape(10, 9) + return z + + +def test_default_async_engine_read_write_roundtrip() -> None: + z = _make_array() + eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + assert isinstance(eng, AsyncArrayEngine) + proto = default_buffer_prototype() + region = Region(start=(2, 1), end_exclusive=(7, 5)) + + out = sync(eng.read_selection(region, prototype=proto)) + np.testing.assert_array_equal(np.asarray(out), np.asarray(z[2:7, 1:5])) + + new: np.ndarray[Any, Any] = np.full((5, 4), -1, dtype="int32") + value = proto.nd_buffer.from_ndarray_like(new) + sync(eng.write_selection(region, value, prototype=proto)) + np.testing.assert_array_equal(np.asarray(z[2:7, 1:5]), new) + + +def test_default_sync_engine_matches_async() -> None: + z = _make_array() + async_eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + eng = DefaultArrayEngine(async_eng) + proto = default_buffer_prototype() + region = Region(start=(0, 0), end_exclusive=(10, 9)) + np.testing.assert_array_equal( + np.asarray(eng.read_selection(region, prototype=proto)), np.asarray(z[:, :]) + ) + + +def test_with_metadata_rebinds() -> None: + z = _make_array() + eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + new_meta = z.async_array.metadata + assert eng.with_metadata(new_meta) is not eng From 8f3d62ce0c4e7c5b95b1846571cb40a8672a5f4c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 15:13:45 +0200 Subject: [PATCH 51/75] fix(engine): capture codec_pipeline results and raise on missing chunks DefaultAsyncArrayEngine.read_selection was discarding the return value of self._codec_pipeline.read(...), preventing the check for missing chunks when config.read_missing_chunks is False. Port the missing chunk detection logic from _get_selection in array.py, capturing results and raising ChunkNotFoundError when missing chunks are encountered and the config flag is False. Assisted-by: ClaudeCode:claude-haiku-4-5 --- src/zarr/core/engine/_default.py | 21 +++++++++++++++++++-- tests/engine/test_default_engine.py | 24 ++++++++++++++++++++++++ 2 files changed, 43 insertions(+), 2 deletions(-) diff --git a/src/zarr/core/engine/_default.py b/src/zarr/core/engine/_default.py index 117f6da19c..e247b0233c 100644 --- a/src/zarr/core/engine/_default.py +++ b/src/zarr/core/engine/_default.py @@ -11,6 +11,7 @@ from zarr.core.common import product from zarr.core.indexing import BasicIndexer from zarr.core.sync import sync +from zarr.errors import ChunkNotFoundError from zarr.storage._common import StorePath if TYPE_CHECKING: @@ -88,7 +89,8 @@ async def read_selection(self, selection: Region, *, prototype: BufferPrototype) if product(indexer.shape) > 0: _config = self._v2_order_config() regular_chunk_spec = self._regular_chunk_spec(_config, prototype) - await self._codec_pipeline.read( + indexed_chunks = list(indexer) + results = await self._codec_pipeline.read( [ ( self.store_path / self.metadata.encode_chunk_key(chunk_coords), @@ -101,11 +103,26 @@ async def read_selection(self, selection: Region, *, prototype: BufferPrototype) out_selection, is_complete_chunk, ) - for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexer + for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexed_chunks ], out_buffer, drop_axes=indexer.drop_axes, ) + if _config.read_missing_chunks is False: + missing_info = [] + for i, result in enumerate(results): + if result["status"] == "missing": + coords = indexed_chunks[i][0] + key = self.metadata.encode_chunk_key(coords) + missing_info.append(f" chunk '{key}' (grid position {coords})") + if missing_info: + chunks_str = "\n".join(missing_info) + raise ChunkNotFoundError( + f"{len(missing_info)} chunk(s) not found in store '{self.store_path}'.\n" + f"Set the 'array.read_missing_chunks' config to True to fill " + f"missing chunks with the fill value.\n" + f"Missing chunks:\n{chunks_str}" + ) return out_buffer.as_ndarray_like() async def write_selection( diff --git a/tests/engine/test_default_engine.py b/tests/engine/test_default_engine.py index 26f2c2a8be..80431c4b7d 100644 --- a/tests/engine/test_default_engine.py +++ b/tests/engine/test_default_engine.py @@ -3,12 +3,14 @@ from typing import Any import numpy as np +import pytest import zarr from zarr.abc.engine import AsyncArrayEngine, Region from zarr.core.buffer import default_buffer_prototype from zarr.core.engine import DefaultArrayEngine, DefaultAsyncArrayEngine from zarr.core.sync import sync +from zarr.errors import ChunkNotFoundError from zarr.storage import MemoryStore @@ -62,3 +64,25 @@ def test_with_metadata_rebinds() -> None: ) new_meta = z.async_array.metadata assert eng.with_metadata(new_meta) is not eng + + +def test_read_missing_chunks_false_raises() -> None: + z = zarr.create_array( + MemoryStore(), + shape=(6,), + chunks=(2,), + dtype="int16", + config={"read_missing_chunks": False}, + ) + z[0:2] = np.arange(2, dtype="int16") # chunks 1 and 2 never written + eng = DefaultAsyncArrayEngine( + store_path=z.async_array.store_path, + metadata=z.async_array.metadata, + config=z.async_array.config, + ) + with pytest.raises(ChunkNotFoundError): + sync( + eng.read_selection( + Region(start=(0,), end_exclusive=(6,)), prototype=default_buffer_prototype() + ) + ) From b74b0c50058e152fbb9ecf56ef41752e8dbe64df Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 15:26:17 +0200 Subject: [PATCH 52/75] feat: engine-spec resolution with per-store hierarchy cache Resolve engine="default"/"zarrista"/None/instance to a bound array engine, caching hierarchy engines per (name, kind, id(store)) so engines minted from the same store share resources. The cache uses a WeakValueDictionary over the hierarchy engine itself, not a plain dict keyed by a weakref/finalize callback on the store as originally sketched: a hierarchy engine holds its store strongly (it needs to, to do I/O), so caching it in a plain dict permanently props up the store's refcount and the finalizer on the store never fires. Minted array engines carry a `_resolve_hierarchy_keepalive` backref so the hierarchy (and its cache entry) lives exactly as long as the engines derived from it, and is dropped once none of them remain. Assisted-by: ClaudeCode:claude-sonnet-5 --- src/zarr/core/engine/__init__.py | 8 ++ src/zarr/core/engine/_resolve.py | 131 +++++++++++++++++++++++++++++++ tests/engine/test_resolve.py | 96 ++++++++++++++++++++++ 3 files changed, 235 insertions(+) create mode 100644 src/zarr/core/engine/_resolve.py create mode 100644 tests/engine/test_resolve.py diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py index ed16e4bbad..4bcf780794 100644 --- a/src/zarr/core/engine/__init__.py +++ b/src/zarr/core/engine/__init__.py @@ -12,16 +12,24 @@ normalize_orthogonal, strip_squeeze, ) +from zarr.core.engine._resolve import ( + EngineName, + resolve_async_engine, + resolve_sync_engine, +) __all__ = [ "DefaultArrayEngine", "DefaultAsyncArrayEngine", "DefaultAsyncHierarchyEngine", "DefaultHierarchyEngine", + "EngineName", "apply_post_index", "normalize_basic", "normalize_block", "normalize_coordinate", "normalize_orthogonal", + "resolve_async_engine", + "resolve_sync_engine", "strip_squeeze", ] diff --git a/src/zarr/core/engine/_resolve.py b/src/zarr/core/engine/_resolve.py new file mode 100644 index 0000000000..e1bc666c29 --- /dev/null +++ b/src/zarr/core/engine/_resolve.py @@ -0,0 +1,131 @@ +"""Resolve an `engine=` argument to a bound array engine.""" + +from __future__ import annotations + +import contextlib +import weakref +from typing import TYPE_CHECKING, Literal + +from zarr.core.engine._default import ( + DefaultAsyncHierarchyEngine, + DefaultHierarchyEngine, +) + +if TYPE_CHECKING: + from collections.abc import Callable + + from zarr.abc.engine import ( + ArrayEngine, + AsyncArrayEngine, + AsyncHierarchyEngine, + HierarchyEngine, + ) + from zarr.abc.store import Store + from zarr.core.metadata import ArrayMetadata + +__all__ = ["EngineName", "resolve_async_engine", "resolve_sync_engine"] + +EngineName = Literal["default", "zarrista"] + +# (name, kind, id(store)) -> hierarchy engine; entries evicted automatically once +# nothing keeps the hierarchy engine itself alive (see `_keepalive` below). +# +# Note: a hierarchy engine holds its `store` strongly (it must, to do I/O), so a +# plain dict keyed by a `weakref.ref`/`weakref.finalize` on the *store* cannot +# work here -- as long as the hierarchy sits in such a cache it keeps the store +# alive, so the store's refcount never reaches zero and the finalizer never +# fires. Using a `WeakValueDictionary` for the hierarchy itself sidesteps that: +# the cache entry disappears as soon as nothing external holds the hierarchy. +# `_keepalive` ties the hierarchy's lifetime to the array engines minted from +# it, so engines resolved for the same store while at least one is still alive +# share a hierarchy; once all of them (and the store) are unreferenced, both +# the hierarchy and the cache entry are collected. +_hierarchy_cache: weakref.WeakValueDictionary[tuple[str, str, int], object] = ( + weakref.WeakValueDictionary() +) + + +def _cached_hierarchy( + name: str, kind: str, store: Store, factory: Callable[[Store], object] +) -> object: + key = (name, kind, id(store)) + hierarchy = _hierarchy_cache.get(key) + if hierarchy is None: + hierarchy = factory(store) + _hierarchy_cache[key] = hierarchy + return hierarchy + + +def _keepalive(engine: object, hierarchy: object) -> object: + """Attach `hierarchy` to `engine` so the hierarchy (and thus the cache entry + tracking it) stays alive for as long as `engine` does. Best-effort: engines + that forbid arbitrary attributes (e.g. via `__slots__`) simply won't share + a cached hierarchy across calls. + """ + with contextlib.suppress(AttributeError): + engine._resolve_hierarchy_keepalive = hierarchy # type: ignore[attr-defined] + return engine + + +def _hierarchy_factory(name: str, *, sync: bool) -> Callable[[Store], object]: + if name == "default": + return DefaultHierarchyEngine if sync else DefaultAsyncHierarchyEngine + if name == "zarrista": + try: + from zarr.zarrista import ( + ZarristaAsyncHierarchyEngine, + ZarristaHierarchyEngine, + ) + except ImportError as e: + raise ImportError( + "engine='zarrista' requires the `zarrista` package; " + "install zarr with the `zarrista` extra" + ) from e + return ( # type: ignore[no-any-return] + ZarristaHierarchyEngine if sync else ZarristaAsyncHierarchyEngine + ) + raise ValueError(f"unknown engine name {name!r}; expected 'default' or 'zarrista'") + + +def resolve_async_engine( + engine: AsyncArrayEngine | EngineName | None, + *, + store: Store, + path: str, + metadata: ArrayMetadata, +) -> AsyncArrayEngine: + """Resolve an `engine=` argument to a bound `AsyncArrayEngine`. + + `None` and `"default"` produce the built-in codec-pipeline engine; + `"zarrista"` lazily imports the `zarrista` package (raising a clear + `ImportError` if it is not installed); an existing engine instance is + returned unchanged. + """ + if engine is None: + engine = "default" + if isinstance(engine, str): + factory = _hierarchy_factory(engine, sync=False) + hierarchy: AsyncHierarchyEngine = _cached_hierarchy( # type: ignore[assignment] + engine, "async", store, factory + ) + return _keepalive(hierarchy.array_engine(path, metadata), hierarchy) # type: ignore[return-value] + return engine + + +def resolve_sync_engine( + engine: ArrayEngine | EngineName | None, + *, + store: Store, + path: str, + metadata: ArrayMetadata, +) -> ArrayEngine: + """Resolve an `engine=` argument to a bound `ArrayEngine`. See `resolve_async_engine`.""" + if engine is None: + engine = "default" + if isinstance(engine, str): + factory = _hierarchy_factory(engine, sync=True) + hierarchy: HierarchyEngine = _cached_hierarchy( # type: ignore[assignment] + engine, "sync", store, factory + ) + return _keepalive(hierarchy.array_engine(path, metadata), hierarchy) # type: ignore[return-value] + return engine diff --git a/tests/engine/test_resolve.py b/tests/engine/test_resolve.py new file mode 100644 index 0000000000..16ccf08bf2 --- /dev/null +++ b/tests/engine/test_resolve.py @@ -0,0 +1,96 @@ +from __future__ import annotations + +import gc +from typing import Any + +import pytest + +import zarr +from zarr.core.engine import ( + DefaultArrayEngine, + DefaultAsyncArrayEngine, + resolve_async_engine, + resolve_sync_engine, +) +from zarr.core.engine._resolve import _hierarchy_cache +from zarr.storage import MemoryStore + + +def _array() -> zarr.Array[Any]: + return zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + + +def test_resolution_combinations() -> None: + z = _array() + store = z.store + path = z.path + meta = z.async_array.metadata + + # None and "default" produce default engines + for spec in (None, "default"): + assert isinstance( + resolve_async_engine(spec, store=store, path=path, metadata=meta), + DefaultAsyncArrayEngine, + ) + assert isinstance( + resolve_sync_engine(spec, store=store, path=path, metadata=meta), + DefaultArrayEngine, + ) + + # instances pass through untouched + inst = resolve_sync_engine(None, store=store, path=path, metadata=meta) + assert resolve_sync_engine(inst, store=store, path=path, metadata=meta) is inst + + # engines minted from the same store share a hierarchy engine + e1 = resolve_async_engine(None, store=store, path=path, metadata=meta) + e2 = resolve_async_engine(None, store=store, path="other", metadata=meta) + assert e1.store_path.store is e2.store_path.store # type: ignore[attr-defined] + + +def test_hierarchy_cache_evicts_when_store_and_engines_are_unreferenced() -> None: + # A dedicated store (not shared with other tests) so the cache starts clean + # for this key. + store = MemoryStore() + z = zarr.create_array(store, shape=(4,), chunks=(2,), dtype="int8") + path = z.path + meta = z.async_array.metadata + + before = len(_hierarchy_cache) + e1 = resolve_async_engine(None, store=store, path=path, metadata=meta) + e2 = resolve_async_engine(None, store=store, path="other", metadata=meta) + assert len(_hierarchy_cache) == before + 1 + + # engines minted while the store is in concurrent use share one hierarchy + assert ( + e1._resolve_hierarchy_keepalive # type: ignore[attr-defined] + is e2._resolve_hierarchy_keepalive # type: ignore[attr-defined] + ) + + del z, e1, e2, store + gc.collect() + assert len(_hierarchy_cache) == before + + +def test_unknown_name_raises() -> None: + z = _array() + with pytest.raises(ValueError, match="unknown engine"): + resolve_async_engine( + "bogus", # type: ignore[arg-type] + store=z.store, + path=z.path, + metadata=z.async_array.metadata, + ) + + +def test_zarrista_missing_raises_import_error() -> None: + try: + import zarrista # noqa: F401 + + pytest.skip("zarrista installed; missing-module error not testable") + except ImportError: + pass + z = _array() + with pytest.raises(ImportError, match="zarrista"): + resolve_async_engine( + "zarrista", store=z.store, path=z.path, metadata=z.async_array.metadata + ) From 5867e3b15f04d89f80b3e98d4b905dfa707836e6 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 15:46:59 +0200 Subject: [PATCH 53/75] feat(engine): thread array config through resolution; add squeeze_axes The array engine must honour the owning array's runtime config (order, read_missing_chunks) rather than re-parsing global defaults. Thread an optional config through resolve_*_engine and the hierarchy array_engine factory; the hierarchy cache stays keyed by store so config differences do not fragment resource sharing. Add squeeze_axes() so orthogonal writes can widen the value with np.newaxis at dropped integer axes (matching oindex_set). Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/abc/engine.py | 9 +++++++-- src/zarr/core/engine/__init__.py | 2 ++ src/zarr/core/engine/_default.py | 12 ++++++++---- src/zarr/core/engine/_normalize.py | 13 +++++++++++++ src/zarr/core/engine/_resolve.py | 12 +++++++++--- 5 files changed, 39 insertions(+), 9 deletions(-) diff --git a/src/zarr/abc/engine.py b/src/zarr/abc/engine.py index 14341f431f..f7ef6260c5 100644 --- a/src/zarr/abc/engine.py +++ b/src/zarr/abc/engine.py @@ -12,6 +12,7 @@ from typing import TYPE_CHECKING, NamedTuple, Protocol, runtime_checkable if TYPE_CHECKING: + from zarr.core.array_spec import ArrayConfig from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer from zarr.core.metadata import ArrayMetadata @@ -81,11 +82,15 @@ def with_metadata(self, metadata: ArrayMetadata) -> AsyncArrayEngine: ... class HierarchyEngine(Protocol): """A store-bound factory for synchronous array engines.""" - def array_engine(self, path: str, metadata: ArrayMetadata) -> ArrayEngine: ... + def array_engine( + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None + ) -> ArrayEngine: ... @runtime_checkable class AsyncHierarchyEngine(Protocol): """A store-bound factory for asynchronous array engines.""" - def array_engine(self, path: str, metadata: ArrayMetadata) -> AsyncArrayEngine: ... + def array_engine( + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None + ) -> AsyncArrayEngine: ... diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py index 4bcf780794..2229111d48 100644 --- a/src/zarr/core/engine/__init__.py +++ b/src/zarr/core/engine/__init__.py @@ -10,6 +10,7 @@ normalize_block, normalize_coordinate, normalize_orthogonal, + squeeze_axes, strip_squeeze, ) from zarr.core.engine._resolve import ( @@ -31,5 +32,6 @@ "normalize_orthogonal", "resolve_async_engine", "resolve_sync_engine", + "squeeze_axes", "strip_squeeze", ] diff --git a/src/zarr/core/engine/_default.py b/src/zarr/core/engine/_default.py index e247b0233c..f7d59e2389 100644 --- a/src/zarr/core/engine/_default.py +++ b/src/zarr/core/engine/_default.py @@ -175,11 +175,13 @@ class DefaultAsyncHierarchyEngine: def __init__(self, store: Store) -> None: self.store = store - def array_engine(self, path: str, metadata: ArrayMetadata) -> DefaultAsyncArrayEngine: + def array_engine( + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None + ) -> DefaultAsyncArrayEngine: return DefaultAsyncArrayEngine( store_path=StorePath(self.store, path), metadata=metadata, - config=parse_array_config(None), + config=config if config is not None else parse_array_config(None), ) @@ -189,5 +191,7 @@ class DefaultHierarchyEngine: def __init__(self, store: Store) -> None: self._async = DefaultAsyncHierarchyEngine(store) - def array_engine(self, path: str, metadata: ArrayMetadata) -> DefaultArrayEngine: - return DefaultArrayEngine(self._async.array_engine(path, metadata)) + def array_engine( + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None + ) -> DefaultArrayEngine: + return DefaultArrayEngine(self._async.array_engine(path, metadata, config)) diff --git a/src/zarr/core/engine/_normalize.py b/src/zarr/core/engine/_normalize.py index 56273f1f1c..cc743cd8a3 100644 --- a/src/zarr/core/engine/_normalize.py +++ b/src/zarr/core/engine/_normalize.py @@ -175,6 +175,19 @@ def strip_squeeze(post: tuple[Any, ...]) -> tuple[Any, ...]: return post +def squeeze_axes(post: tuple[Any, ...]) -> tuple[int, ...]: + """Return the axes recorded by a trailing `_Squeeze` marker, else `()`. + + Orthogonal *writes* assign into the unsqueezed box using + `strip_squeeze(post)`; when the selection dropped integer axes, the value + must be widened with `np.newaxis` at exactly these axes (matching + `oindex_set`). Returns `()` when `post` carries no marker. + """ + if post and isinstance(post[-1], _Squeeze): + return post[-1].axes + return () + + def normalize_coordinate( selection: tuple[Any, ...], shape: tuple[int, ...] ) -> tuple[Region, tuple[np.ndarray, ...]]: diff --git a/src/zarr/core/engine/_resolve.py b/src/zarr/core/engine/_resolve.py index e1bc666c29..f831f5aed7 100644 --- a/src/zarr/core/engine/_resolve.py +++ b/src/zarr/core/engine/_resolve.py @@ -21,6 +21,7 @@ HierarchyEngine, ) from zarr.abc.store import Store + from zarr.core.array_spec import ArrayConfig from zarr.core.metadata import ArrayMetadata __all__ = ["EngineName", "resolve_async_engine", "resolve_sync_engine"] @@ -93,13 +94,17 @@ def resolve_async_engine( store: Store, path: str, metadata: ArrayMetadata, + config: ArrayConfig | None = None, ) -> AsyncArrayEngine: """Resolve an `engine=` argument to a bound `AsyncArrayEngine`. `None` and `"default"` produce the built-in codec-pipeline engine; `"zarrista"` lazily imports the `zarrista` package (raising a clear `ImportError` if it is not installed); an existing engine instance is - returned unchanged. + returned unchanged. `config`, when given, is threaded to the engine so it + honours the owning array's runtime configuration (e.g. `order`, + `read_missing_chunks`); the hierarchy cache is keyed only by store, so + engines for arrays with differing configs still share resources. """ if engine is None: engine = "default" @@ -108,7 +113,7 @@ def resolve_async_engine( hierarchy: AsyncHierarchyEngine = _cached_hierarchy( # type: ignore[assignment] engine, "async", store, factory ) - return _keepalive(hierarchy.array_engine(path, metadata), hierarchy) # type: ignore[return-value] + return _keepalive(hierarchy.array_engine(path, metadata, config), hierarchy) # type: ignore[return-value] return engine @@ -118,6 +123,7 @@ def resolve_sync_engine( store: Store, path: str, metadata: ArrayMetadata, + config: ArrayConfig | None = None, ) -> ArrayEngine: """Resolve an `engine=` argument to a bound `ArrayEngine`. See `resolve_async_engine`.""" if engine is None: @@ -127,5 +133,5 @@ def resolve_sync_engine( hierarchy: HierarchyEngine = _cached_hierarchy( # type: ignore[assignment] engine, "sync", store, factory ) - return _keepalive(hierarchy.array_engine(path, metadata), hierarchy) # type: ignore[return-value] + return _keepalive(hierarchy.array_engine(path, metadata, config), hierarchy) # type: ignore[return-value] return engine From 474fee07d01374435c278d1afbe2321c6778196a Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 16:38:22 +0200 Subject: [PATCH 54/75] feat: AsyncArray routes selection I/O through its engine AsyncArray gains an `engine: AsyncArrayEngine` field, resolved in __init__ (threading the array's config) and rebound via with_metadata on resize/append. All selection methods (basic/orthogonal/coordinate/mask/ block, get and set) now normalize to a `(Region, post_index)` pair and route through the engine instead of the codec pipeline: - module-level _get_selection reads the box and post-indexes it (scalar extraction, out= copy, fields, custom NDArrayLike preservation for identity reads). - _set_selection broadcasts the value into a full-box write, else does a facade-level read-modify-write (orthogonal _Squeeze widening included). - historical indexer validation (BasicIndexer/OrthogonalIndexer/ CoordinateIndexer error messages, negative-step and bad-type rejection) is preserved in _basic/_orthogonal/_coordinate_region_post helpers. The default engine is lazily imported inside _default.py to break the array<->engine import cycle. test_accessed_chunks and one resolve cache test are updated to the engine's documented bounding-box access semantics (per the design spec's accepted consequence); the internal _get_selection call in test_with_data uses the public getitem API. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 1030 +++++++++++------------- src/zarr/core/engine/_default.py | 10 +- tests/engine/test_asyncarray_wiring.py | 60 ++ tests/engine/test_resolve.py | 9 +- tests/test_array.py | 7 +- tests/test_indexing.py | 62 +- 6 files changed, 584 insertions(+), 594 deletions(-) create mode 100644 tests/engine/test_asyncarray_wiring.py diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index d15c70064b..d4083370a7 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -21,6 +21,7 @@ import zarr from zarr.abc.codec import ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec, Codec +from zarr.abc.engine import Region from zarr.abc.numcodec import Numcodec, _is_numcodec from zarr.codecs._v2 import V2Codec from zarr.codecs.bytes import BytesCodec @@ -81,32 +82,47 @@ parse_dtype, ) from zarr.core.dtype.common import HasEndianness, HasItemSize, HasObjectCodec +from zarr.core.engine import ( + apply_post_index, + normalize_basic, + normalize_coordinate, + normalize_orthogonal, + resolve_async_engine, + squeeze_axes, + strip_squeeze, +) from zarr.core.indexing import ( AsyncOIndex, AsyncVIndex, - BasicIndexer, BasicSelection, BlockIndex, BlockIndexer, - CoordinateIndexer, CoordinateSelection, Fields, - Indexer, - MaskIndexer, MaskSelection, OIndex, - OrthogonalIndexer, OrthogonalSelection, Selection, VIndex, _iter_grid, _iter_regions, + boundscheck_indices, check_fields, check_no_multi_fields, + ensure_tuple, + is_bool_array, + is_coordinate_selection, + is_integer, + is_integer_array, + is_mask_selection, is_pure_fancy_indexing, is_pure_orthogonal_indexing, is_scalar, + normalize_integer_selection, pop_fields, + replace_ellipsis, + replace_lists, + wraparound_indices, ) from zarr.core.metadata import ( ArrayMetadata, @@ -131,8 +147,8 @@ from zarr.core.sync import sync from zarr.errors import ( ArrayNotFoundError, - ChunkNotFoundError, MetadataValidationError, + NegativeStepError, ZarrDeprecationWarning, ZarrUserWarning, ) @@ -152,9 +168,11 @@ import numpy.typing as npt from zarr.abc.codec import CodecPipeline + from zarr.abc.engine import AsyncArrayEngine from zarr.abc.store import Store from zarr.codecs.sharding import ShardingCodecIndexLocation from zarr.core.dtype.wrapper import TBaseDType, TBaseScalar + from zarr.core.engine import EngineName from zarr.storage import StoreLike from zarr.types import AnyArray, AnyAsyncArray, ArrayV2, ArrayV3, AsyncArrayV2, AsyncArrayV3 @@ -327,6 +345,9 @@ class AsyncArray[T_ArrayMetadata: (ArrayV2Metadata, ArrayV3Metadata)]: metadata: T_ArrayMetadata store_path: StorePath codec_pipeline: CodecPipeline = field(init=False) + # derived, per-array data path; excluded from equality/repr like other + # engine instances lack value semantics (two default engines are never `==`) + engine: AsyncArrayEngine = field(init=False, compare=False, repr=False) _chunk_grid: ChunkGrid = field(init=False) config: ArrayConfig @@ -336,6 +357,7 @@ def __init__( metadata: ArrayV2Metadata | ArrayV2MetadataDict, store_path: StorePath, config: ArrayConfigLike | None = None, + engine: AsyncArrayEngine | EngineName | None = None, ) -> None: ... @overload @@ -344,6 +366,7 @@ def __init__( metadata: ArrayV3Metadata | ArrayMetadataJSON_V3, store_path: StorePath, config: ArrayConfigLike | None = None, + engine: AsyncArrayEngine | EngineName | None = None, ) -> None: ... def __init__( @@ -351,6 +374,7 @@ def __init__( metadata: ArrayMetadata | ArrayMetadataDict, store_path: StorePath, config: ArrayConfigLike | None = None, + engine: AsyncArrayEngine | EngineName | None = None, ) -> None: metadata_parsed = parse_array_metadata(metadata) config_parsed = parse_array_config(config) @@ -364,6 +388,17 @@ def __init__( "codec_pipeline", create_codec_pipeline(metadata=metadata_parsed, store=store_path.store), ) + object.__setattr__( + self, + "engine", + resolve_async_engine( + engine, + store=store_path.store, + path=store_path.path, + metadata=metadata_parsed, + config=config_parsed, + ), + ) @classmethod async def _create( @@ -1411,22 +1446,25 @@ def nbytes(self) -> int: async def _get_selection( self, - indexer: Indexer, + region: Region, + post_index: tuple[Any, ...], *, prototype: BufferPrototype, out: NDBuffer | None = None, fields: Fields | None = None, + scalarize: bool = False, ) -> NDArrayLikeOrScalar: + """Route a normalized `(region, post_index)` read through this array's engine.""" return await _get_selection( - self.store_path, + self.engine, self.metadata, - self.codec_pipeline, self.config, - self._chunk_grid, - indexer, + region, + post_index, prototype=prototype, out=out, fields=fields, + scalarize=scalarize, ) async def getitem( @@ -1471,15 +1509,10 @@ async def getitem( np.int32(0) """ - return await _getitem( - self.store_path, - self.metadata, - self.codec_pipeline, - self.config, - self._chunk_grid, - selection, - prototype=prototype, - ) + if prototype is None: + prototype = default_buffer_prototype() + region, post = _basic_region_post(selection, self.metadata.shape) + return await self._get_selection(region, post, prototype=prototype, scalarize=True) async def get_orthogonal_selection( self, @@ -1489,17 +1522,10 @@ async def get_orthogonal_selection( fields: Fields | None = None, prototype: BufferPrototype | None = None, ) -> NDArrayLikeOrScalar: - return await _get_orthogonal_selection( - self.store_path, - self.metadata, - self.codec_pipeline, - self.config, - self._chunk_grid, - selection, - out=out, - fields=fields, - prototype=prototype, - ) + if prototype is None: + prototype = default_buffer_prototype() + region, post = _orthogonal_region_post(selection, self.metadata.shape) + return await self._get_selection(region, post, prototype=prototype, out=out, fields=fields) async def get_mask_selection( self, @@ -1509,17 +1535,10 @@ async def get_mask_selection( fields: Fields | None = None, prototype: BufferPrototype | None = None, ) -> NDArrayLikeOrScalar: - return await _get_mask_selection( - self.store_path, - self.metadata, - self.codec_pipeline, - self.config, - self._chunk_grid, - mask, - out=out, - fields=fields, - prototype=prototype, - ) + if prototype is None: + prototype = default_buffer_prototype() + region, post = _mask_region_post(mask, self.metadata.shape) + return await self._get_selection(region, post, prototype=prototype, out=out, fields=fields) async def get_coordinate_selection( self, @@ -1529,17 +1548,16 @@ async def get_coordinate_selection( fields: Fields | None = None, prototype: BufferPrototype | None = None, ) -> NDArrayLikeOrScalar: - return await _get_coordinate_selection( - self.store_path, - self.metadata, - self.codec_pipeline, - self.config, - self._chunk_grid, - selection, - out=out, - fields=fields, - prototype=prototype, + if prototype is None: + prototype = default_buffer_prototype() + region, post, sel_shape = _coordinate_region_post(selection, self.metadata.shape) + out_array = await self._get_selection( + region, post, prototype=prototype, out=out, fields=fields ) + if hasattr(out_array, "shape"): + # restore the (possibly multi-dimensional) selection shape + out_array = cast("NDArrayLikeOrScalar", np.asarray(out_array).reshape(sel_shape)) + return out_array async def _save_metadata(self, metadata: ArrayMetadata, ensure_parents: bool = False) -> None: """ @@ -1549,19 +1567,20 @@ async def _save_metadata(self, metadata: ArrayMetadata, ensure_parents: bool = F async def _set_selection( self, - indexer: Indexer, + region: Region, + post_index: tuple[Any, ...], value: npt.ArrayLike, *, prototype: BufferPrototype, fields: Fields | None = None, ) -> None: + """Route a normalized `(region, post_index)` write through this array's engine.""" return await _set_selection( - self.store_path, + self.engine, self.metadata, - self.codec_pipeline, self.config, - self._chunk_grid, - indexer, + region, + post_index, value, prototype=prototype, fields=fields, @@ -1606,16 +1625,10 @@ async def setitem( - This method is asynchronous and should be awaited. - Supports basic indexing, where the selection is contiguous and does not involve advanced indexing. """ - return await _setitem( - self.store_path, - self.metadata, - self.codec_pipeline, - self.config, - self._chunk_grid, - selection, - value, - prototype=prototype, - ) + if prototype is None: + prototype = default_buffer_prototype() + region, post = _basic_region_post(selection, self.metadata.shape) + return await self._set_selection(region, post, value, prototype=prototype) @property def oindex(self) -> AsyncOIndex[T_ArrayMetadata]: @@ -2826,12 +2839,15 @@ def get_basic_selection( if prototype is None: prototype = default_buffer_prototype() + region, post = _basic_region_post(selection, self.shape) return sync( self.async_array._get_selection( - BasicIndexer(selection, self.shape, self._chunk_grid), + region, + post, out=out, fields=fields, prototype=prototype, + scalarize=True, ) ) @@ -2935,8 +2951,10 @@ def set_basic_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = BasicIndexer(selection, self.shape, self._chunk_grid) - sync(self.async_array._set_selection(indexer, value, fields=fields, prototype=prototype)) + region, post = _basic_region_post(selection, self.shape) + sync( + self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + ) def get_orthogonal_selection( self, @@ -3063,10 +3081,10 @@ def get_orthogonal_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = OrthogonalIndexer(selection, self.shape, self._chunk_grid) + region, post = _orthogonal_region_post(selection, self.shape) return sync( self.async_array._get_selection( - indexer=indexer, out=out, fields=fields, prototype=prototype + region, post, out=out, fields=fields, prototype=prototype ) ) @@ -3181,9 +3199,9 @@ def set_orthogonal_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = OrthogonalIndexer(selection, self.shape, self._chunk_grid) + region, post = _orthogonal_region_post(selection, self.shape) return sync( - self.async_array._set_selection(indexer, value, fields=fields, prototype=prototype) + self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) ) def get_mask_selection( @@ -3269,10 +3287,10 @@ def get_mask_selection( if prototype is None: prototype = default_buffer_prototype() - indexer = MaskIndexer(mask, self.shape, self._chunk_grid) + region, post = _mask_region_post(mask, self.shape) return sync( self.async_array._get_selection( - indexer=indexer, out=out, fields=fields, prototype=prototype + region, post, out=out, fields=fields, prototype=prototype ) ) @@ -3358,8 +3376,10 @@ def set_mask_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = MaskIndexer(mask, self.shape, self._chunk_grid) - sync(self.async_array._set_selection(indexer, value, fields=fields, prototype=prototype)) + region, post = _mask_region_post(mask, self.shape) + sync( + self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + ) def get_coordinate_selection( self, @@ -3446,16 +3466,16 @@ def get_coordinate_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = CoordinateIndexer(selection, self.shape, self._chunk_grid) + region, post, sel_shape = _coordinate_region_post(selection, self.shape) out_array = sync( self.async_array._get_selection( - indexer=indexer, out=out, fields=fields, prototype=prototype + region, post, out=out, fields=fields, prototype=prototype ) ) if hasattr(out_array, "shape"): # restore shape - out_array = np.array(out_array).reshape(indexer.sel_shape) + out_array = np.asarray(out_array).reshape(sel_shape) return out_array def set_coordinate_selection( @@ -3537,8 +3557,9 @@ def set_coordinate_selection( """ if prototype is None: prototype = default_buffer_prototype() - # setup indexer - indexer = CoordinateIndexer(selection, self.shape, self._chunk_grid) + # normalize the coordinate selection to a contiguous box + pointwise index + region, post, sel_shape = _coordinate_region_post(selection, self.shape) + flat_shape = (int(product(sel_shape)),) # handle value - need ndarray-like flatten value if not is_scalar(value, self.dtype): @@ -3553,14 +3574,16 @@ def set_coordinate_selection( value = np.array(value).reshape(-1) if not is_scalar(value, self.dtype) and ( - isinstance(value, NDArrayLike) and indexer.shape != value.shape + isinstance(value, NDArrayLike) and flat_shape != value.shape ): raise ValueError( - f"Attempting to set a selection of {indexer.sel_shape[0]} " + f"Attempting to set a selection of {sel_shape[0]} " f"elements with an array of {value.shape[0]} elements." ) - sync(self.async_array._set_selection(indexer, value, fields=fields, prototype=prototype)) + sync( + self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + ) def get_block_selection( self, @@ -3659,11 +3682,9 @@ def get_block_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = BlockIndexer(selection, self.shape, self._chunk_grid) + region = _block_region(selection, self.shape, self._chunk_grid) return sync( - self.async_array._get_selection( - indexer=indexer, out=out, fields=fields, prototype=prototype - ) + self.async_array._get_selection(region, (), out=out, fields=fields, prototype=prototype) ) def set_block_selection( @@ -3759,8 +3780,8 @@ def set_block_selection( """ if prototype is None: prototype = default_buffer_prototype() - indexer = BlockIndexer(selection, self.shape, self._chunk_grid) - sync(self.async_array._set_selection(indexer, value, fields=fields, prototype=prototype)) + region = _block_region(selection, self.shape, self._chunk_grid) + sync(self.async_array._set_selection(region, (), value, fields=fields, prototype=prototype)) @property def vindex(self) -> VIndex: @@ -5350,520 +5371,395 @@ def _get_chunk_spec( ) -async def _get_selection( - store_path: StorePath, - metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, - config: ArrayConfig, - chunk_grid: ChunkGrid, - indexer: Indexer, - *, - prototype: BufferPrototype, - out: NDBuffer | None = None, - fields: Fields | None = None, -) -> NDArrayLikeOrScalar: - """ - Get a selection from an array. +def _native_dtype(metadata: ArrayMetadata) -> np.dtype[Any]: + """The array's native numpy dtype, for both Zarr v2 and v3 metadata.""" + zdtype = metadata.dtype if metadata.zarr_format == 2 else metadata.data_type + return zdtype.to_native_dtype() - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - indexer : Indexer - The indexer specifying the selection. - prototype : BufferPrototype - A buffer prototype to use for the retrieved data. - out : NDBuffer | None, optional - An output buffer to write the data to. - fields : Fields | None, optional - Fields to select from structured arrays. - - Returns - ------- - NDArrayLikeOrScalar - The selected data. - """ - # Get dtype from metadata - if metadata.zarr_format == 2: - zdtype = metadata.dtype - else: - zdtype = metadata.data_type - dtype = zdtype.to_native_dtype() - # Determine memory order - if metadata.zarr_format == 2: - order = metadata.order - else: - order = config.order - - # check fields are sensible - out_dtype = check_fields(fields, dtype) +def _selection_result_shape( + region: Region, post_index: tuple[Any, ...], out_dtype: np.dtype[Any] +) -> tuple[int, ...]: + """Shape of the post-indexed result for an ndim-preserving box read.""" + return apply_post_index(np.empty(region.shape, dtype=out_dtype), post_index).shape - # setup output buffer - if out is not None: - if isinstance(out, NDBuffer): - out_buffer = out - else: - raise TypeError(f"out argument needs to be an NDBuffer. Got {type(out)!r}") - if out_buffer.shape != indexer.shape: - raise ValueError( - f"shape of out argument doesn't match. Expected {indexer.shape}, got {out.shape}" - ) - else: - out_buffer = prototype.nd_buffer.empty( - shape=indexer.shape, - dtype=out_dtype, - order=order, - ) - if product(indexer.shape) > 0: - # need to use the order from the metadata for v2 - _config = config - if metadata.zarr_format == 2: - _config = replace(_config, order=order) - - # reading chunks and decoding them - indexed_chunks = list(indexer) - # For regular grids, all chunks share the same ArraySpec, so build it once - # and reuse it to avoid per-chunk ChunkGrid lookups and ArraySpec construction. - regular_grid = chunk_grid.is_regular - if regular_grid: - regular_chunk_spec = ArraySpec( - shape=chunk_grid.chunk_shape, - dtype=metadata.dtype, - fill_value=metadata.fill_value, - config=_config, - prototype=prototype, - ) - results = await codec_pipeline.read( - [ - ( - store_path / metadata.encode_chunk_key(chunk_coords), - regular_chunk_spec - if regular_grid - else _get_chunk_spec(metadata, chunk_grid, chunk_coords, _config, prototype), - chunk_selection, - out_selection, - is_complete_chunk, - ) - for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexed_chunks - ], - out_buffer, - drop_axes=indexer.drop_axes, - ) - if _config.read_missing_chunks is False: - missing_info = [] - for i, result in enumerate(results): - if result["status"] == "missing": - coords = indexed_chunks[i][0] - key = metadata.encode_chunk_key(coords) - missing_info.append(f" chunk '{key}' (grid position {coords})") - if missing_info: - chunks_str = "\n".join(missing_info) - raise ChunkNotFoundError( - f"{len(missing_info)} chunk(s) not found in store '{store_path}'.\n" - f"Set the 'array.read_missing_chunks' config to True to fill " - f"missing chunks with the fill value.\n" - f"Missing chunks:\n{chunks_str}" - ) - if isinstance(indexer, BasicIndexer) and indexer.shape == (): - return out_buffer.as_scalar() - return out_buffer.as_ndarray_like() +def _axis_covers_full_ordered(p: Any, n: int, axis: int, ndim: int) -> bool: + """Whether post entry `p` selects axis `axis` (length `n`) fully and in order. -async def _getitem( - store_path: StorePath, - metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, - config: ArrayConfig, - chunk_grid: ChunkGrid, - selection: BasicSelection, - *, - prototype: BufferPrototype | None = None, -) -> NDArrayLikeOrScalar: + True for a full-range step-1 slice, and for an `np.ix_` outer-product array + for this axis -- shape `n` at `axis` and 1 elsewhere, values `arange(n)`. A + pointwise (coordinate) array has a different shape (`(N,)` on every axis), so + it is correctly rejected: `box[([0,1,2],[0,1,2])]` is the diagonal, not the + whole box. """ - Retrieve a subset of the array's data based on the provided selection. - - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - chunk_grid : ChunkGrid - The chunk grid. - selection : BasicSelection - A selection object specifying the subset of data to retrieve. - prototype : BufferPrototype, optional - A buffer prototype to use for the retrieved data (default is None). - - Returns - ------- - NDArrayLikeOrScalar - The retrieved subset of the array's data. + if isinstance(p, slice): + return p.start is None and p.stop is None and p.step in (None, 1) + if isinstance(p, np.ndarray): + expected_shape = tuple(n if j == axis else 1 for j in range(ndim)) + return p.shape == expected_shape and np.array_equal(p.reshape(-1), np.arange(n)) + return False + + +def _is_identity_read(post_index: tuple[Any, ...], box_shape: tuple[int, ...]) -> bool: + """Whether the post index leaves the ndim-preserving box read unchanged. + + True for an empty post (block selection), a per-axis tuple of full step-1 + slices (contiguous basic selections), and full-axis orthogonal outer slices + (`np.ix_(arange(n))`). In those cases the box the engine returned already *is* + the result, so it is returned without numpy coercion, preserving a custom + `NDArrayLike` type. Integer axes and the `_Squeeze` marker drop dimensions, so + they make the length differ or fail the per-axis check and are excluded. """ - if prototype is None: - prototype = default_buffer_prototype() - indexer = BasicIndexer( - selection, - shape=metadata.shape, - chunk_grid=chunk_grid, - ) - return await _get_selection( - store_path, metadata, codec_pipeline, config, chunk_grid, indexer, prototype=prototype + if post_index == (): + return True + if len(post_index) != len(box_shape): + return False + ndim = len(box_shape) + return all( + _axis_covers_full_ordered(p, n, i, ndim) + for i, (p, n) in enumerate(zip(post_index, box_shape, strict=True)) ) -async def _get_orthogonal_selection( - store_path: StorePath, - metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, - config: ArrayConfig, - chunk_grid: ChunkGrid, - selection: OrthogonalSelection, - *, - out: NDBuffer | None = None, - fields: Fields | None = None, - prototype: BufferPrototype | None = None, +def _finalize_result( + result: NDArrayLike, out: NDBuffer | None, *, scalarize: bool ) -> NDArrayLikeOrScalar: - """ - Get an orthogonal selection from the array. - - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - chunk_grid : ChunkGrid - The chunk grid. - selection : OrthogonalSelection - The orthogonal selection specification. - out : NDBuffer | None, optional - An output buffer to write the data to. - fields : Fields | None, optional - Fields to select from structured arrays. - prototype : BufferPrototype | None, optional - A buffer prototype to use for the retrieved data. - - Returns - ------- - NDArrayLikeOrScalar - The selected data. - """ - if prototype is None: - prototype = default_buffer_prototype() - indexer = OrthogonalIndexer(selection, metadata.shape, chunk_grid) - return await _get_selection( - store_path, - metadata, - codec_pipeline, - config, - chunk_grid, - indexer=indexer, - out=out, - fields=fields, - prototype=prototype, - ) + """Apply the shared post-read finalization: `out` copy, or scalar extraction.""" + if out is not None: + out.as_ndarray_like()[...] = result # type: ignore[index] + return out.as_ndarray_like() + if scalarize and result.shape == (): + # basic indexing collapses to a scalar (matches the historical + # `out_buffer.as_scalar()` return for all-integer basic selections) + return cast("NDArrayLikeOrScalar", np.asarray(result)[()]) + return result -async def _get_mask_selection( - store_path: StorePath, +async def _get_selection( + engine: AsyncArrayEngine, metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, config: ArrayConfig, - chunk_grid: ChunkGrid, - mask: MaskSelection, + region: Region, + post_index: tuple[Any, ...], *, + prototype: BufferPrototype, out: NDBuffer | None = None, fields: Fields | None = None, - prototype: BufferPrototype | None = None, + scalarize: bool = False, ) -> NDArrayLikeOrScalar: - """ - Get a mask selection from the array. - - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - chunk_grid : ChunkGrid - The chunk grid. - mask : MaskSelection - The boolean mask specifying the selection. - out : NDBuffer | None, optional - An output buffer to write the data to. - fields : Fields | None, optional - Fields to select from structured arrays. - prototype : BufferPrototype | None, optional - A buffer prototype to use for the retrieved data. - - Returns - ------- - NDArrayLikeOrScalar - The selected data. - """ - if prototype is None: - prototype = default_buffer_prototype() - indexer = MaskIndexer(mask, metadata.shape, chunk_grid) - return await _get_selection( - store_path, - metadata, - codec_pipeline, - config, - chunk_grid, - indexer=indexer, - out=out, - fields=fields, - prototype=prototype, - ) + """Read a contiguous `region` via the engine and re-index it to a selection. - -async def _get_coordinate_selection( - store_path: StorePath, - metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, - config: ArrayConfig, - chunk_grid: ChunkGrid, - selection: CoordinateSelection, - *, - out: NDBuffer | None = None, - fields: Fields | None = None, - prototype: BufferPrototype | None = None, -) -> NDArrayLikeOrScalar: + The engine only speaks contiguous boxes; `normalize_*` produced `region` + (the box to transfer) and `post_index` (the numpy index that turns the + ndim-preserving box read into the requested selection). This applies + `fields`, then `post_index`, and finally optional scalar extraction / `out` + handling. """ - Get a coordinate selection from the array. + dtype = _native_dtype(metadata) + out_dtype = check_fields(fields, dtype) + result_shape = _selection_result_shape(region, post_index, out_dtype) - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - chunk_grid : ChunkGrid - The chunk grid. - selection : CoordinateSelection - The coordinate selection specification. - out : NDBuffer | None, optional - An output buffer to write the data to. - fields : Fields | None, optional - Fields to select from structured arrays. - prototype : BufferPrototype | None, optional - A buffer prototype to use for the retrieved data. + if out is not None: + if not isinstance(out, NDBuffer): + raise TypeError(f"out argument needs to be an NDBuffer. Got {type(out)!r}") + if out.shape != result_shape: + raise ValueError( + f"shape of out argument doesn't match. Expected {result_shape}, got {out.shape}" + ) - Returns - ------- - NDArrayLikeOrScalar - The selected data. + if product(region.shape) == 0: + empty = np.empty(result_shape, dtype=out_dtype) + return _finalize_result(empty, out, scalarize=scalarize) + + raw = await engine.read_selection(region, prototype=prototype) + if not fields and _is_identity_read(post_index, region.shape): + # a full-box, step-1 read is exactly the engine result; return it + # unchanged so a custom `NDArrayLike` type (e.g. GPU/torch buffers) + # survives instead of being coerced to numpy by `np.asarray` + return _finalize_result(raw, out, scalarize=scalarize) + box = np.asarray(raw) + if fields: + # non-empty `fields` selects structured sub-fields; an empty list/tuple + # (from `pop_fields` on a field-free selection) means "all fields" + box = box[fields] # type: ignore[index] + result = apply_post_index(box, post_index) + return _finalize_result(result, out, scalarize=scalarize) + + +def _coerce_write_value( + value: npt.ArrayLike, dtype: np.dtype[Any], prototype: BufferPrototype, fields: Fields | None +) -> NDArrayLike: + """Coerce a user-supplied write `value` to an ndarray-like of the array dtype. + + Ported from the historical `_set_selection`: scalars are materialized with + the prototype's buffer type (so GPU prototypes stay on-device), and + array-likes are cast to the array dtype. When `fields` is set the value + keeps its own (structured sub-field) dtype. """ - if prototype is None: - prototype = default_buffer_prototype() - indexer = CoordinateIndexer(selection, metadata.shape, chunk_grid) - out_array = await _get_selection( - store_path, - metadata, - codec_pipeline, - config, - chunk_grid, - indexer=indexer, - out=out, - fields=fields, - prototype=prototype, - ) - - if hasattr(out_array, "shape"): - # restore shape - out_array = cast("NDArrayLikeOrScalar", np.array(out_array).reshape(indexer.sel_shape)) - return out_array + # empty `fields` (list/tuple) means "no field selection"; treat it like None + target_dtype = None if fields else dtype + if np.isscalar(value): + array_like = prototype.buffer.create_zero_length().as_array_like() + # only NDArrayLike implements __array_function__ (e.g. numpy, cupy) + if isinstance(array_like, np._typing._SupportsArrayFunc): + array_like_ = cast("np._typing._SupportsArrayFunc", array_like) + value = np.asanyarray(value, dtype=target_dtype, like=array_like_) + else: + if not hasattr(value, "shape"): + value = np.asarray(value, dtype=target_dtype) + if target_dtype is not None and ( + not hasattr(value, "dtype") or value.dtype.name != dtype.name + ): + if hasattr(value, "astype"): + # Handle things that are already NDArrayLike more efficiently + value = value.astype(dtype=dtype, order="A") + else: + value = np.array(value, dtype=dtype, order="A") + return cast("NDArrayLike", value) async def _set_selection( - store_path: StorePath, + engine: AsyncArrayEngine, metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, config: ArrayConfig, - chunk_grid: ChunkGrid, - indexer: Indexer, + region: Region, + post_index: tuple[Any, ...], value: npt.ArrayLike, *, prototype: BufferPrototype, fields: Fields | None = None, ) -> None: + """Write a selection through the engine as a contiguous-box read-modify-write. + + An identity `post_index` (a full-box write with no `fields`) broadcasts the + value straight into the box and writes it. Any strided/fancy/orthogonal/ + fields write instead reads the box, patches it with numpy using the same + `post_index` the read path uses (dropping the orthogonal `_Squeeze` marker + and widening the value at dropped integer axes, matching `oindex_set`), then + writes the whole box back. """ - Set a selection in an array. - - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - chunk_grid : ChunkGrid - The chunk grid. - indexer : Indexer - The indexer specifying the selection. - value : npt.ArrayLike - The values to write. - prototype : BufferPrototype - A buffer prototype to use. - fields : Fields | None, optional - Fields to select from structured arrays. - """ - # Get dtype from metadata - if metadata.zarr_format == 2: - zdtype = metadata.dtype - else: - zdtype = metadata.data_type - dtype = zdtype.to_native_dtype() - - # check fields are sensible + dtype = _native_dtype(metadata) check_fields(fields, dtype) fields = check_no_multi_fields(fields) - - # check value shape - if np.isscalar(value): - array_like = prototype.buffer.create_zero_length().as_array_like() - if isinstance(array_like, np._typing._SupportsArrayFunc): - # TODO: need to handle array types that don't support __array_function__ - # like PyTorch and JAX - array_like_ = cast("np._typing._SupportsArrayFunc", array_like) - value = np.asanyarray(value, dtype=dtype, like=array_like_) + if product(region.shape) == 0: + return + + value_np = _coerce_write_value(value, dtype, prototype, fields) + + stripped = strip_squeeze(post_index) + sq_axes = squeeze_axes(post_index) + # a "full box" write addresses every element of the box exactly once, in + # order, so the value can be broadcast straight in without a read. This + # covers contiguous basic slices (post `slice(None, None, 1)`), full-axis + # orthogonal slices (post `np.ix_(arange(n))`), integer axes (length-1 + # boxes), and block selections (post `()`). + identity_post = not fields and _is_full_box_write(stripped, region.shape) + + # orthogonal selections drop integer axes; widen the value with np.newaxis at + # those axes so it aligns with the (un-squeezed) box (matching `oindex_set`) + assign_value = _widen_value_for_squeeze(value_np, sq_axes, len(region.shape)) + + if identity_post: + # full-box write: broadcast the value into the box without a read + box = np.array(np.broadcast_to(np.asarray(assign_value), region.shape), dtype=dtype) else: - if not hasattr(value, "shape"): - value = np.asarray(value, dtype) - # assert ( - # value.shape == indexer.shape - # ), f"shape of value doesn't match indexer shape. Expected {indexer.shape}, got {value.shape}" - if not hasattr(value, "dtype") or value.dtype.name != dtype.name: - if hasattr(value, "astype"): - # Handle things that are already NDArrayLike more efficiently - value = value.astype(dtype=dtype, order="A") - else: - value = np.array(value, dtype=dtype, order="A") - value = cast("NDArrayLike", value) + # facade-level read-modify-write for strided/fancy/orthogonal/fields writes + box = np.array(np.asarray(await engine.read_selection(region, prototype=prototype))) + target = box[fields] if fields else box # type: ignore[index] + if stripped == (): + target[...] = assign_value + else: + target[stripped] = assign_value - # We accept any ndarray like object from the user and convert it - # to an NDBuffer (or subclass). From this point onwards, we only pass - # Buffer and NDBuffer between components. - value_buffer = prototype.nd_buffer.from_ndarray_like(value) + # `np.ascontiguousarray` would promote a 0-d box (scalar array) to shape (1,); + # a 0-d array is already contiguous, so pass it through unchanged. + contiguous_box = box if box.ndim == 0 else np.ascontiguousarray(box) + value_buffer = prototype.nd_buffer.from_ndarray_like(contiguous_box) + await engine.write_selection(region, value_buffer, prototype=prototype) - # Determine memory order - if metadata.zarr_format == 2: - order = metadata.order - else: - order = config.order - - # need to use the order from the metadata for v2 - _config = config - if metadata.zarr_format == 2: - _config = replace(_config, order=order) - - # merging with existing data and encoding chunks - # For regular grids, all chunks share the same ArraySpec, so build it once - # and reuse it to avoid per-chunk ChunkGrid lookups and ArraySpec construction. - regular_grid = chunk_grid.is_regular - if regular_grid: - regular_chunk_spec = ArraySpec( - shape=chunk_grid.chunk_shape, - dtype=metadata.dtype, - fill_value=metadata.fill_value, - config=_config, - prototype=prototype, - ) - await codec_pipeline.write( - [ - ( - store_path / metadata.encode_chunk_key(chunk_coords), - regular_chunk_spec - if regular_grid - else _get_chunk_spec(metadata, chunk_grid, chunk_coords, _config, prototype), - chunk_selection, - out_selection, - is_complete_chunk, + +def _basic_region_post( + selection: BasicSelection, shape: tuple[int, ...] +) -> tuple[Region, tuple[slice | int, ...]]: + """Validate a basic selection with the historical rules, then normalize it. + + `normalize_basic` (the engine helper) accepts negative-step slices and raises + `TypeError` for non-basic elements, but the public zarr basic-indexing API + has always rejected both with `IndexError` (via `BasicIndexer`). This + re-imposes those rules -- too-many-indices, negative steps, and unsupported + element types -- before delegating the box computation to `normalize_basic`. + """ + sel = replace_ellipsis(selection, shape) + for dim_sel, dim_len in zip(sel, shape, strict=True): + if is_integer(dim_sel): + normalize_integer_selection(dim_sel, dim_len) + elif isinstance(dim_sel, slice): + if dim_sel.step is not None and dim_sel.step < 1: + raise NegativeStepError("only slices with step >= 1 are supported.") + else: + raise IndexError( + "unsupported selection item for basic indexing; " + f"expected integer or slice, got {type(dim_sel)!r}" ) - for chunk_coords, chunk_selection, out_selection, is_complete_chunk in indexer - ], - value_buffer, - drop_axes=indexer.drop_axes, - ) + return normalize_basic(cast("BasicSelection", sel), shape) -async def _setitem( - store_path: StorePath, - metadata: ArrayMetadata, - codec_pipeline: CodecPipeline, - config: ArrayConfig, - chunk_grid: ChunkGrid, - selection: BasicSelection, - value: npt.ArrayLike, - prototype: BufferPrototype | None = None, -) -> None: +def _orthogonal_region_post( + selection: OrthogonalSelection, shape: tuple[int, ...] +) -> tuple[Region, tuple[Any, ...]]: + """Validate an orthogonal selection with the historical rules, then normalize it. + + Mirrors `OrthogonalIndexer`'s per-axis validation (and its exact error + messages) -- integer bounds, negative-step slices, 1-d integer/boolean + arrays of the right length, and unsupported element types -- before handing + the box computation to `normalize_orthogonal`. """ - Set values in the array using basic indexing. + sel = replace_lists(replace_ellipsis(selection, shape)) + for dim_sel, dim_len in zip(sel, shape, strict=True): + if is_integer(dim_sel): + normalize_integer_selection(dim_sel, dim_len) + elif isinstance(dim_sel, slice): + if dim_sel.step is not None and dim_sel.step < 1: + raise NegativeStepError("only slices with step >= 1 are supported.") + elif is_integer_array(dim_sel): + if not is_integer_array(dim_sel, 1): + raise IndexError( + "integer arrays in an orthogonal selection must be 1-dimensional only" + ) + checked = np.asanyarray(dim_sel).copy() + wraparound_indices(checked, dim_len) + boundscheck_indices(checked, dim_len) + elif is_bool_array(dim_sel): + if not is_bool_array(dim_sel, 1): + raise IndexError( + "Boolean arrays in an orthogonal selection must be 1-dimensional only" + ) + if dim_sel.shape[0] != dim_len: + raise IndexError( + f"Boolean array has the wrong length for dimension; expected {dim_len}, " + f"got {dim_sel.shape[0]}" + ) + else: + raise IndexError( + "unsupported selection item for orthogonal indexing; " + "expected integer, slice, integer array or Boolean " + f"array, got {type(dim_sel)!r}" + ) + return normalize_orthogonal(sel, shape) - Parameters - ---------- - store_path : StorePath - The store path of the array. - metadata : ArrayMetadata - The array metadata. - codec_pipeline : CodecPipeline - The codec pipeline for encoding/decoding. - config : ArrayConfig - The array configuration. - chunk_grid : ChunkGrid - The chunk grid. - selection : BasicSelection - The selection defining the region of the array to set. - value : npt.ArrayLike - The values to be written into the selected region of the array. - prototype : BufferPrototype or None, optional - A prototype buffer that defines the structure and properties of the array chunks being modified. - If None, the default buffer prototype is used. + +def _widen_value_for_squeeze( + value: NDArrayLike, squeeze_axes_: tuple[int, ...], ndim: int +) -> NDArrayLike: + """Insert `np.newaxis` at dropped integer axes so a value aligns with the box. + + Orthogonal selections drop integer axes from the result, so a non-scalar + value has fewer dimensions than the ndim-preserving box. Re-inserting a + size-1 axis at each dropped position lets the value broadcast/assign against + the box, matching `zarr.core.indexing.oindex_set`. Scalars and 0-d values are + returned unchanged (they broadcast on their own). """ - if prototype is None: - prototype = default_buffer_prototype() - indexer = BasicIndexer( - selection, - shape=metadata.shape, - chunk_grid=chunk_grid, - ) - return await _set_selection( - store_path, - metadata, - codec_pipeline, - config, - chunk_grid, - indexer, - value, - prototype=prototype, - ) + if squeeze_axes_ and not np.isscalar(value) and np.asarray(value).ndim > 0: + value_selection: list[Any] = [slice(None)] * ndim + for ax in squeeze_axes_: + value_selection[ax] = np.newaxis + return cast("NDArrayLike", np.asarray(value)[tuple(value_selection)]) + return value + + +def _is_full_box_write(stripped: tuple[Any, ...], box_shape: tuple[int, ...]) -> bool: + """Whether a write's post index addresses every box element exactly once, in order. + + When true, the value fills the whole box and the write needs no + read-modify-write. Handles the post-index forms the `normalize_*` helpers + emit: an empty tuple (block selection), step-1 basic slices, single integer + axes (length-1 boxes), and `np.ix_` arrays equal to `arange(n)` (full-axis + orthogonal slices). The trailing `_Squeeze` marker must already be removed. + """ + if stripped == (): + return True + if len(stripped) != len(box_shape): + return False + ndim = len(box_shape) + for i, (p, n) in enumerate(zip(stripped, box_shape, strict=True)): + if isinstance(p, (int, np.integer)): + # an integer axis writes a length-1 box axis fully + if n != 1: + return False + elif not _axis_covers_full_ordered(p, n, i, ndim): + return False + return True + + +def _block_region( + selection: BasicSelection, shape: tuple[int, ...], chunk_grid: ChunkGrid +) -> Region: + """Map a block (chunk-grid) selection to its contiguous element-space box. + + Block selections are always contiguous (a slice of whole chunks spans a + contiguous element range), so the post index is empty. `BlockIndexer` is + reused purely for its coordinate math, preserving every existing behavior: + integer and step-1 slice block indices, irregular (rectilinear) grids, and + bounds errors. + """ + indexer = BlockIndexer(selection, shape, chunk_grid) + starts = tuple(di.start for di in indexer.dim_indexers) + ends = tuple(di.stop for di in indexer.dim_indexers) + return Region(start=starts, end_exclusive=ends) + + +def _coordinate_region_post( + selection: CoordinateSelection, shape: tuple[int, ...] +) -> tuple[Region, tuple[np.ndarray[Any, Any], ...], tuple[int, ...]]: + """Normalize a coordinate (vindex) selection to `(region, post, sel_shape)`. + + Replicates `CoordinateIndexer`'s normalization and validation: integer axes + are widened to length-1 arrays, lists become arrays, the selection is checked + to be one integer array per dimension (raising the same `IndexError`), and + per-axis wraparound / bounds checks are applied (raising the same "dimension + with length" `IndexError`). The per-axis arrays are then broadcast to a common + shape (`sel_shape`) and flattened to the pointwise index the engine box read + is re-indexed with; the caller reshapes the flat result back to `sel_shape`. + """ + sel = ensure_tuple(selection) + sel = tuple(np.asarray([i]) if is_integer(i) else i for i in sel) + sel = replace_lists(sel) + if not is_coordinate_selection(sel, shape): + raise IndexError( + "invalid coordinate selection; expected one integer " + "(coordinate) array per dimension of the target array, " + f"got {selection!r}" + ) + for dim_sel, dim_len in zip(sel, shape, strict=True): + checked = np.asanyarray(dim_sel).copy() + wraparound_indices(checked, dim_len) + boundscheck_indices(checked, dim_len) + broadcast = np.broadcast_arrays(*(np.asarray(s) for s in sel)) + sel_shape = broadcast[0].shape or (1,) + flat = tuple(np.reshape(b, -1) for b in broadcast) + region, post = normalize_coordinate(flat, shape) + return region, post, sel_shape + + +def _mask_region_post( + mask: MaskSelection, shape: tuple[int, ...] +) -> tuple[Region, tuple[np.ndarray[Any, Any], ...]]: + """Normalize a boolean-mask (vindex) selection to `(region, post)`. + + Equivalent to coordinate indexing: the mask is converted to coordinate + arrays with `np.nonzero`, matching `MaskIndexer`. The result is 1-d in + row-major order, so no reshape is needed afterwards. + """ + sel = ensure_tuple(mask) + sel = replace_lists(sel) + if not is_mask_selection(sel, shape): + raise IndexError( + "invalid mask selection; expected one Boolean (mask)" + f"array with the same shape as the target array, got {sel!r}" + ) + coords = np.nonzero(sel[0]) + return normalize_coordinate(coords, shape) async def _resize( @@ -5913,9 +5809,10 @@ async def _delete_key(key: str) -> None: # Write new metadata await save_metadata(array.store_path, new_metadata) - # Update metadata and chunk_grid (in place) + # Update metadata, chunk_grid, and engine (in place) object.__setattr__(array, "metadata", new_metadata) object.__setattr__(array, "_chunk_grid", new_chunk_grid) + object.__setattr__(array, "engine", array.engine.with_metadata(new_metadata)) async def _append( @@ -5976,14 +5873,15 @@ async def _append( slice(None) if i != axis else slice(old_shape[i], new_shape[i]) for i in range(len(array.shape)) ) - await _setitem( - array.store_path, + region, post = _basic_region_post(append_selection, array.metadata.shape) + await _set_selection( + array.engine, array.metadata, - array.codec_pipeline, array.config, - array._chunk_grid, - append_selection, + region, + post, data, + prototype=default_buffer_prototype(), ) return new_shape diff --git a/src/zarr/core/engine/_default.py b/src/zarr/core/engine/_default.py index f7d59e2389..a3078c3a5c 100644 --- a/src/zarr/core/engine/_default.py +++ b/src/zarr/core/engine/_default.py @@ -5,7 +5,6 @@ from dataclasses import replace from typing import TYPE_CHECKING -from zarr.core.array import _get_chunk_spec, create_codec_pipeline from zarr.core.array_spec import ArraySpec, parse_array_config from zarr.core.chunk_grids import ChunkGrid from zarr.core.common import product @@ -37,6 +36,11 @@ class DefaultAsyncArrayEngine: """Codec-pipeline-backed engine. Any store, Zarr v2 and v3.""" def __init__(self, store_path: StorePath, metadata: ArrayMetadata, config: ArrayConfig) -> None: + # Imported lazily to avoid an import cycle: `zarr.core.array` imports the + # engine package (for engine resolution) while the default engine reuses + # `zarr.core.array`'s codec-pipeline machinery. + from zarr.core.array import create_codec_pipeline + self.store_path = store_path self.metadata = metadata self.config = config @@ -78,6 +82,8 @@ def _v2_order_config(self) -> ArrayConfig: return self.config async def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + from zarr.core.array import _get_chunk_spec + indexer = self._indexer(selection) if self.metadata.zarr_format == 2: dtype = self.metadata.dtype.to_native_dtype() @@ -128,6 +134,8 @@ async def read_selection(self, selection: Region, *, prototype: BufferPrototype) async def write_selection( self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype ) -> None: + from zarr.core.array import _get_chunk_spec + indexer = self._indexer(selection) _config = self._v2_order_config() regular_chunk_spec = self._regular_chunk_spec(_config, prototype) diff --git a/tests/engine/test_asyncarray_wiring.py b/tests/engine/test_asyncarray_wiring.py new file mode 100644 index 0000000000..0e6295c38b --- /dev/null +++ b/tests/engine/test_asyncarray_wiring.py @@ -0,0 +1,60 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import numpy as np + +import zarr +from zarr.abc.engine import Region +from zarr.core.engine import DefaultAsyncArrayEngine +from zarr.storage import MemoryStore + +if TYPE_CHECKING: + from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer + from zarr.core.metadata import ArrayMetadata + + +class _SpyEngine: + """Wraps a real engine, recording regions.""" + + def __init__(self, inner: DefaultAsyncArrayEngine) -> None: + self.inner = inner + self.read_regions: list[Region] = [] + self.write_regions: list[Region] = [] + + async def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + self.read_regions.append(selection) + return await self.inner.read_selection(selection, prototype=prototype) + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + self.write_regions.append(selection) + return await self.inner.write_selection(selection, value, prototype=prototype) + + def with_metadata(self, metadata: ArrayMetadata) -> _SpyEngine: + return _SpyEngine(self.inner.with_metadata(metadata)) + + +async def test_asyncarray_routes_io_through_engine() -> None: + # NOTE: the async variant of the spy test. Task 6 gives the sync `Array` its + # own sync engine; until then the sync `z[...]` path shares this async engine, + # but this task exercises the async `AsyncArray` methods directly. + z = zarr.create_array(MemoryStore(), shape=(10,), chunks=(3,), dtype="int16") + aa = z.async_array + spy = _SpyEngine( + DefaultAsyncArrayEngine(store_path=aa.store_path, metadata=aa.metadata, config=aa.config) + ) + object.__setattr__(aa, "engine", spy) + + await aa.setitem(slice(2, 8), np.arange(6, dtype="int16")) + data = await aa.getitem(slice(2, 8)) + + assert spy.write_regions == [Region(start=(2,), end_exclusive=(8,))] + assert spy.read_regions == [Region(start=(2,), end_exclusive=(8,))] + np.testing.assert_array_equal(np.asarray(data), np.arange(6, dtype="int16")) + + +def test_asyncarray_default_engine_attribute() -> None: + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + assert isinstance(z.async_array.engine, DefaultAsyncArrayEngine) diff --git a/tests/engine/test_resolve.py b/tests/engine/test_resolve.py index 16ccf08bf2..32bddc5fab 100644 --- a/tests/engine/test_resolve.py +++ b/tests/engine/test_resolve.py @@ -49,13 +49,18 @@ def test_resolution_combinations() -> None: def test_hierarchy_cache_evicts_when_store_and_engines_are_unreferenced() -> None: # A dedicated store (not shared with other tests) so the cache starts clean - # for this key. + # for this key. Measure the baseline *before* creating the array: creating it + # resolves the array's own engine, which already populates the (default, + # async, id(store)) cache entry that `e1`/`e2` then reuse. store = MemoryStore() + # Collect any hierarchies left unreferenced by earlier tests first, so the + # baseline reflects only entries kept alive by still-referenced arrays. + gc.collect() + before = len(_hierarchy_cache) z = zarr.create_array(store, shape=(4,), chunks=(2,), dtype="int8") path = z.path meta = z.async_array.metadata - before = len(_hierarchy_cache) e1 = resolve_async_engine(None, store=store, path=path, metadata=meta) e2 = resolve_async_engine(None, store=store, path="other", metadata=meta) assert len(_hierarchy_cache) == before + 1 diff --git a/tests/test_array.py b/tests/test_array.py index 0d6d2d5906..109119e315 100644 --- a/tests/test_array.py +++ b/tests/test_array.py @@ -70,7 +70,7 @@ from zarr.core.dtype.common import ENDIANNESS_STR, EndiannessStr from zarr.core.dtype.npy.common import NUMPY_ENDIANNESS_STR, endianness_from_numpy_str from zarr.core.group import AsyncGroup -from zarr.core.indexing import BasicIndexer, _iter_grid, _iter_regions +from zarr.core.indexing import _iter_grid, _iter_regions from zarr.core.metadata.v2 import ArrayV2Metadata from zarr.core.sync import sync from zarr.errors import ( @@ -1596,10 +1596,7 @@ async def test_with_data(impl: Literal["sync", "async"], store: Store) -> None: stored = arr[:] elif impl == "async": arr = await create_array(store, name=name, data=data, zarr_format=3) - stored = await arr._get_selection( - BasicIndexer(..., shape=arr.shape, chunk_grid=arr._chunk_grid), - prototype=default_buffer_prototype(), - ) + stored = await arr.getitem(Ellipsis, prototype=default_buffer_prototype()) else: raise ValueError(f"Invalid impl: {impl}") diff --git a/tests/test_indexing.py b/tests/test_indexing.py index a9358e4fcf..ee0e6b7580 100644 --- a/tests/test_indexing.py +++ b/tests/test_indexing.py @@ -1821,15 +1821,33 @@ async def test_accessed_chunks( z = zarr_array_from_numpy_array(StorePath(store), np.zeros(shape), chunk_shape=chunks) for ii, (optype, slices) in enumerate(ops): - # Resolve the slices into the accessed chunks for each dimension - chunks_per_dim = [] + # The array engine interchanges selections as contiguous, step-1 `Region` + # boxes: a selection is served through its step-1 *bounding box*, + # deliberately replacing the old per-chunk gather/scatter (see + # docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md, + # "Consequence, accepted for now"). + # + # Reads fetch the whole bounding box, so a strided slice reads every chunk + # between its first and last selected index. A write is a bounding-box + # read-modify-write: it reads the box, patches the selected elements, and + # writes the box back -- but `write_empty_chunks=False` (the default) + # drops box chunks that stay all-fill, so only chunks that actually + # contain a selected element are written. Model the two sets separately. + box_chunks_per_dim = [] # every chunk the step-1 bounding box overlaps + selected_chunks_per_dim = [] # chunks containing an actually selected index for N, C, sl in zip(shape, chunks, slices, strict=True): - chunk_ind = np.arange(N, dtype=int)[sl] // C - chunks_per_dim.append(np.unique(chunk_ind)) + selected = np.arange(N, dtype=int)[sl] + selected_chunks_per_dim.append(np.unique(selected // C)) + if selected.size: + bounding_box = np.arange(selected.min(), selected.max() + 1) + else: + bounding_box = selected + box_chunks_per_dim.append(np.unique(bounding_box // C)) - # Combine and generate the cartesian product to determine the chunks keys that - # will be accessed - chunks_accessed = [".".join(map(str, comb)) for comb in itertools.product(*chunks_per_dim)] + box_chunks = [".".join(map(str, comb)) for comb in itertools.product(*box_chunks_per_dim)] + selected_chunks = [ + ".".join(map(str, comb)) for comb in itertools.product(*selected_chunks_per_dim) + ] counts_before = store.counter.copy() @@ -1837,24 +1855,28 @@ async def test_accessed_chunks( if optype == "__getitem__": z[slices] else: + # a non-zero fill value keeps the selected chunks non-empty so they + # are written (ii == 0 only for the very first op, which is a read) z[slices] = ii # Get the change in counts delta_counts = store.counter - counts_before - # Check that the access counts for the operation have increased by one for all - # the chunks we expect to be included - for ci in chunks_accessed: - assert delta_counts.pop((optype, ci)) == 1 - - # If the chunk was partially written to it will also have been read once. We - # don't determine if the chunk was actually partial here, just that the - # counts are consistent that this might have happened - if optype == "__setitem__": - assert ("__getitem__", ci) not in delta_counts or delta_counts.pop( - ("__getitem__", ci) - ) == 1 - # Check that no other chunks were accessed + if optype == "__getitem__": + # a read fetches the whole bounding box, and nothing else + for ci in box_chunks: + assert delta_counts.pop(("__getitem__", ci)) == 1 + else: + # the RMW reads bounding-box chunks (unless the write covers the whole + # box, in which case there is no facade read); an edge chunk only + # partially covered by the box is read again by the codec pipeline's + # partial-chunk merge, so tolerate more than one read per box chunk. + for ci in box_chunks: + delta_counts.pop(("__getitem__", ci), None) + # ...and writes back the chunks that end up non-empty + for ci in selected_chunks: + assert delta_counts.pop(("__setitem__", ci)) >= 1 + # Check that no chunks outside the bounding box / selection were accessed assert len(delta_counts) == 0 From f5c90df9a488dc35d2def88981977be4f0527046 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 16:55:23 +0200 Subject: [PATCH 55/75] fix(engine): preserve memory order on strided/fancy reads The engine-based module-level _get_selection lost the array's effective memory order on non-identity reads: apply_post_index ran on a bare np.asarray(raw), always yielding a C-contiguous result even when the array's effective order is F (v2 metadata.order / v3 config.order). The identity/full-box path already returned the engine result unchanged and was unaffected. Materialize the post-indexed result with np.asarray(result, order=order) using the same order derivation already used elsewhere (AsyncArray.order), restoring the pre-engine behavior where the out buffer was allocated with the effective order up front. Assisted-by: ClaudeCode:claude-sonnet-5 --- src/zarr/core/array.py | 2 ++ tests/engine/test_asyncarray_wiring.py | 16 ++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index d4083370a7..87502bdb6c 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -5459,6 +5459,7 @@ async def _get_selection( dtype = _native_dtype(metadata) out_dtype = check_fields(fields, dtype) result_shape = _selection_result_shape(region, post_index, out_dtype) + order = metadata.order if metadata.zarr_format == 2 else config.order if out is not None: if not isinstance(out, NDBuffer): @@ -5484,6 +5485,7 @@ async def _get_selection( # (from `pop_fields` on a field-free selection) means "all fields" box = box[fields] # type: ignore[index] result = apply_post_index(box, post_index) + result = np.asarray(result, order=order) return _finalize_result(result, out, scalarize=scalarize) diff --git a/tests/engine/test_asyncarray_wiring.py b/tests/engine/test_asyncarray_wiring.py index 0e6295c38b..43c1530016 100644 --- a/tests/engine/test_asyncarray_wiring.py +++ b/tests/engine/test_asyncarray_wiring.py @@ -58,3 +58,19 @@ async def test_asyncarray_routes_io_through_engine() -> None: def test_asyncarray_default_engine_attribute() -> None: z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") assert isinstance(z.async_array.engine, DefaultAsyncArrayEngine) + + +def test_strided_read_preserves_fortran_order() -> None: + z = zarr.create_array( + MemoryStore(), + shape=(8, 8), + chunks=(4, 4), + dtype="float64", + config={"order": "F"}, + ) + z[:, :] = np.asfortranarray(np.arange(64, dtype="float64").reshape(8, 8)) + full = np.asarray(z[:, :]) + strided = np.asarray(z[::2, ::2]) + assert full.flags.f_contiguous + assert strided.flags.f_contiguous + np.testing.assert_array_equal(strided, np.arange(64.0).reshape(8, 8)[::2, ::2]) From 6e8c68dd5cd464d1ea664de5af0b4f86f8ca9d65 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 17:32:12 +0200 Subject: [PATCH 56/75] feat: Array sync data path calls its engine without the event loop `Array` now lazily resolves its own `ArrayEngine` (cached on `_engine`, seeded from an optional `engine_spec` constructor arg) instead of routing every read/write through `sync(self.async_array._get_selection(...))`. All ten get/set-selection call sites route through the new module-level `_get_selection_sync`/`_set_selection_sync`, which share their pure logic (dtype/order/`out` validation, identity-read detection, RMW box patching, result finalization) with the async `_get_selection`/`_set_selection` via `_get_selection_prepare`/`_finish_get_selection` and `_prepare_set_selection`/`_patch_selection_box`/`_finish_set_selection`, so the async and sync variants differ only in awaiting the two engine calls. `resize`/`append` rebind the cached engine to post-mutation metadata via a new `_rebind_engine` helper, so reads/writes past the old bounds don't hit a stale engine. Assisted-by: ClaudeCode:claude-sonnet-5 --- src/zarr/core/array.py | 431 ++++++++++++++++++++----- tests/engine/test_asyncarray_wiring.py | 7 +- tests/engine/test_sync_path.py | 82 +++++ 3 files changed, 439 insertions(+), 81 deletions(-) create mode 100644 tests/engine/test_sync_path.py diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 87502bdb6c..63e46ef141 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -3,13 +3,14 @@ import warnings from asyncio import gather from collections.abc import Iterable, Mapping, Sequence -from dataclasses import dataclass, field, replace +from dataclasses import InitVar, dataclass, field, replace from itertools import starmap from logging import getLogger from typing import ( TYPE_CHECKING, Any, Literal, + NamedTuple, TypedDict, cast, overload, @@ -88,6 +89,7 @@ normalize_coordinate, normalize_orthogonal, resolve_async_engine, + resolve_sync_engine, squeeze_axes, strip_squeeze, ) @@ -168,7 +170,7 @@ import numpy.typing as npt from zarr.abc.codec import CodecPipeline - from zarr.abc.engine import AsyncArrayEngine + from zarr.abc.engine import ArrayEngine, AsyncArrayEngine from zarr.abc.store import Store from zarr.codecs.sharding import ShardingCodecIndexLocation from zarr.core.dtype.wrapper import TBaseDType, TBaseScalar @@ -1814,6 +1816,18 @@ class Array[T_ArrayMetadata: (ArrayV2Metadata, ArrayV3Metadata)]: """ _async_array: AsyncArray[T_ArrayMetadata] + # `engine_spec` seeds `_engine`'s lazy resolution (see the `engine` property + # below); it is not itself a stored attribute (`InitVar`), so it cannot + # share the `engine` property's name without the class-body assignment of + # one clobbering the other in `Array.__dict__`. + engine_spec: InitVar[ArrayEngine | EngineName | None] = None + # derived, per-array sync data path; lazily resolved on first `.engine` + # access (see the `engine` property) so wrapping an `AsyncArray` never + # eagerly binds a second engine that might go unused. + _engine: ArrayEngine | None = field(default=None, init=False, repr=False, compare=False) + + def __post_init__(self, engine_spec: ArrayEngine | EngineName | None) -> None: + self._engine_spec = engine_spec @property def async_array(self) -> AsyncArray[T_ArrayMetadata]: @@ -1825,6 +1839,39 @@ def async_array(self) -> AsyncArray[T_ArrayMetadata]: """ return self._async_array + @property + def engine(self) -> ArrayEngine: + """The synchronous array engine serving this array's data path. + + Resolved lazily on first access and cached; every `get_*_selection`/ + `set_*_selection` method (and `__getitem__`/`__setitem__`) calls the + engine directly, without wrapping the call in a coroutine, so this + array's data path never touches the event loop in the caller's thread + -- the default engine still uses `sync()` internally, but that runs on + zarr's dedicated loop thread, not the caller's. + """ + if self._engine is None: + aa = self._async_array + self._engine = resolve_sync_engine( + self._engine_spec, + store=aa.store_path.store, + path=aa.store_path.path, + metadata=aa.metadata, + config=aa.config, + ) + return self._engine + + def _rebind_engine(self) -> None: + """Rebind the cached sync engine to this array's current metadata. + + Called after `resize`/`append` mutate `self._async_array.metadata` in + place, so a subsequent read/write through `self.engine` sees the new + shape instead of a stale cached engine. A no-op if `.engine` was never + accessed (nothing cached to rebind). + """ + if self._engine is not None: + self._engine = self._engine.with_metadata(self._async_array.metadata) + @property def config(self) -> ArrayConfig: """ @@ -2840,15 +2887,16 @@ def get_basic_selection( if prototype is None: prototype = default_buffer_prototype() region, post = _basic_region_post(selection, self.shape) - return sync( - self.async_array._get_selection( - region, - post, - out=out, - fields=fields, - prototype=prototype, - scalarize=True, - ) + return _get_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + out=out, + fields=fields, + prototype=prototype, + scalarize=True, ) def set_basic_selection( @@ -2952,8 +3000,15 @@ def set_basic_selection( if prototype is None: prototype = default_buffer_prototype() region, post = _basic_region_post(selection, self.shape) - sync( - self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + _set_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + value, + fields=fields, + prototype=prototype, ) def get_orthogonal_selection( @@ -3082,10 +3137,15 @@ def get_orthogonal_selection( if prototype is None: prototype = default_buffer_prototype() region, post = _orthogonal_region_post(selection, self.shape) - return sync( - self.async_array._get_selection( - region, post, out=out, fields=fields, prototype=prototype - ) + return _get_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + out=out, + fields=fields, + prototype=prototype, ) def set_orthogonal_selection( @@ -3200,8 +3260,15 @@ def set_orthogonal_selection( if prototype is None: prototype = default_buffer_prototype() region, post = _orthogonal_region_post(selection, self.shape) - return sync( - self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + _set_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + value, + fields=fields, + prototype=prototype, ) def get_mask_selection( @@ -3288,10 +3355,15 @@ def get_mask_selection( if prototype is None: prototype = default_buffer_prototype() region, post = _mask_region_post(mask, self.shape) - return sync( - self.async_array._get_selection( - region, post, out=out, fields=fields, prototype=prototype - ) + return _get_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + out=out, + fields=fields, + prototype=prototype, ) def set_mask_selection( @@ -3377,8 +3449,15 @@ def set_mask_selection( if prototype is None: prototype = default_buffer_prototype() region, post = _mask_region_post(mask, self.shape) - sync( - self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + _set_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + value, + fields=fields, + prototype=prototype, ) def get_coordinate_selection( @@ -3467,10 +3546,15 @@ def get_coordinate_selection( if prototype is None: prototype = default_buffer_prototype() region, post, sel_shape = _coordinate_region_post(selection, self.shape) - out_array = sync( - self.async_array._get_selection( - region, post, out=out, fields=fields, prototype=prototype - ) + out_array = _get_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + out=out, + fields=fields, + prototype=prototype, ) if hasattr(out_array, "shape"): @@ -3581,8 +3665,15 @@ def set_coordinate_selection( f"elements with an array of {value.shape[0]} elements." ) - sync( - self.async_array._set_selection(region, post, value, fields=fields, prototype=prototype) + _set_selection_sync( + self.engine, + self.metadata, + self.config, + region, + post, + value, + fields=fields, + prototype=prototype, ) def get_block_selection( @@ -3683,8 +3774,15 @@ def get_block_selection( if prototype is None: prototype = default_buffer_prototype() region = _block_region(selection, self.shape, self._chunk_grid) - return sync( - self.async_array._get_selection(region, (), out=out, fields=fields, prototype=prototype) + return _get_selection_sync( + self.engine, + self.metadata, + self.config, + region, + (), + out=out, + fields=fields, + prototype=prototype, ) def set_block_selection( @@ -3781,7 +3879,16 @@ def set_block_selection( if prototype is None: prototype = default_buffer_prototype() region = _block_region(selection, self.shape, self._chunk_grid) - sync(self.async_array._set_selection(region, (), value, fields=fields, prototype=prototype)) + _set_selection_sync( + self.engine, + self.metadata, + self.config, + region, + (), + value, + fields=fields, + prototype=prototype, + ) @property def vindex(self) -> VIndex: @@ -3846,6 +3953,7 @@ def resize(self, new_shape: ShapeLike) -> None: ``` """ sync(self.async_array.resize(new_shape)) + self._rebind_engine() def append(self, data: npt.ArrayLike, axis: int = 0) -> tuple[int, ...]: """Append `data` to `axis`. @@ -3881,7 +3989,9 @@ def append(self, data: npt.ArrayLike, axis: int = 0) -> tuple[int, ...]: >>> z.shape (20000, 2000) """ - return sync(self.async_array.append(data, axis=axis)) + new_shape = sync(self.async_array.append(data, axis=axis)) + self._rebind_engine() + return new_shape def update_attributes(self, new_attributes: dict[str, JSON]) -> Self: """ @@ -5436,25 +5546,20 @@ def _finalize_result( return result -async def _get_selection( - engine: AsyncArrayEngine, +def _get_selection_prepare( metadata: ArrayMetadata, config: ArrayConfig, region: Region, post_index: tuple[Any, ...], *, - prototype: BufferPrototype, - out: NDBuffer | None = None, - fields: Fields | None = None, - scalarize: bool = False, -) -> NDArrayLikeOrScalar: - """Read a contiguous `region` via the engine and re-index it to a selection. - - The engine only speaks contiguous boxes; `normalize_*` produced `region` - (the box to transfer) and `post_index` (the numpy index that turns the - ndim-preserving box read into the requested selection). This applies - `fields`, then `post_index`, and finally optional scalar extraction / `out` - handling. + out: NDBuffer | None, + fields: Fields | None, +) -> tuple[np.dtype[Any], tuple[int, ...], MemoryOrder]: + """Shared preamble for `_get_selection`/`_get_selection_sync`: compute the + read's output dtype/shape/memory order and validate `out` against them. + + Split out so the async and sync selection helpers share this pure logic + and differ only in whether `engine.read_selection` is awaited. """ dtype = _native_dtype(metadata) out_dtype = check_fields(fields, dtype) @@ -5468,12 +5573,24 @@ async def _get_selection( raise ValueError( f"shape of out argument doesn't match. Expected {result_shape}, got {out.shape}" ) + return out_dtype, result_shape, order - if product(region.shape) == 0: - empty = np.empty(result_shape, dtype=out_dtype) - return _finalize_result(empty, out, scalarize=scalarize) - raw = await engine.read_selection(region, prototype=prototype) +def _finish_get_selection( + raw: NDArrayLike, + region: Region, + post_index: tuple[Any, ...], + *, + fields: Fields | None, + order: MemoryOrder, + out: NDBuffer | None, + scalarize: bool, +) -> NDArrayLikeOrScalar: + """Shared postamble for `_get_selection`/`_get_selection_sync`: apply + `fields`/`post_index` to the engine's raw box read, then finalize into + `out`/scalar -- everything that happens after the (a)waited + `engine.read_selection` call. + """ if not fields and _is_identity_read(post_index, region.shape): # a full-box, step-1 read is exactly the engine result; return it # unchanged so a custom `NDArrayLike` type (e.g. GPU/torch buffers) @@ -5489,6 +5606,68 @@ async def _get_selection( return _finalize_result(result, out, scalarize=scalarize) +async def _get_selection( + engine: AsyncArrayEngine, + metadata: ArrayMetadata, + config: ArrayConfig, + region: Region, + post_index: tuple[Any, ...], + *, + prototype: BufferPrototype, + out: NDBuffer | None = None, + fields: Fields | None = None, + scalarize: bool = False, +) -> NDArrayLikeOrScalar: + """Read a contiguous `region` via the engine and re-index it to a selection. + + The engine only speaks contiguous boxes; `normalize_*` produced `region` + (the box to transfer) and `post_index` (the numpy index that turns the + ndim-preserving box read into the requested selection). This applies + `fields`, then `post_index`, and finally optional scalar extraction / `out` + handling. + """ + out_dtype, result_shape, order = _get_selection_prepare( + metadata, config, region, post_index, out=out, fields=fields + ) + if product(region.shape) == 0: + empty = np.empty(result_shape, dtype=out_dtype) + return _finalize_result(empty, out, scalarize=scalarize) + + raw = await engine.read_selection(region, prototype=prototype) + return _finish_get_selection( + raw, region, post_index, fields=fields, order=order, out=out, scalarize=scalarize + ) + + +def _get_selection_sync( + engine: ArrayEngine, + metadata: ArrayMetadata, + config: ArrayConfig, + region: Region, + post_index: tuple[Any, ...], + *, + prototype: BufferPrototype, + out: NDBuffer | None = None, + fields: Fields | None = None, + scalarize: bool = False, +) -> NDArrayLikeOrScalar: + """Synchronous mirror of `_get_selection`: same logic, calling + `engine.read_selection` directly instead of awaiting it, so `Array`'s data + path never runs a coroutine on the caller's thread. + """ + out_dtype, result_shape, order = _get_selection_prepare( + metadata, config, region, post_index, out=out, fields=fields + ) + if product(region.shape) == 0: + empty = np.empty(result_shape, dtype=out_dtype) + return _finalize_result(empty, out, scalarize=scalarize) + + raw = engine.read_selection(region, prototype=prototype) + return _finish_get_selection( + raw, region, post_index, fields=fields, order=order, out=out, scalarize=scalarize + ) + + def _coerce_write_value( value: npt.ArrayLike, dtype: np.dtype[Any], prototype: BufferPrototype, fields: Fields | None ) -> NDArrayLike: @@ -5521,31 +5700,42 @@ def _coerce_write_value( return cast("NDArrayLike", value) -async def _set_selection( - engine: AsyncArrayEngine, +class _SetSelectionPrep(NamedTuple): + """Engine-agnostic state prepared for a selection write, shared by + `_set_selection`/`_set_selection_sync`. + + `identity_box`, when not `None`, is the fully-assembled box for a + full-box overwrite -- no read is needed, it is ready to write as-is. When + `None`, the write is a read-modify-write: `fields`/`stripped`/ + `assign_value` are the arguments `_patch_selection_box` needs to patch the + box the engine reads back. + """ + + identity_box: NDArrayLike | None + fields: Fields | None + stripped: tuple[Any, ...] + assign_value: NDArrayLike + + +def _prepare_set_selection( metadata: ArrayMetadata, - config: ArrayConfig, region: Region, post_index: tuple[Any, ...], value: npt.ArrayLike, *, prototype: BufferPrototype, - fields: Fields | None = None, -) -> None: - """Write a selection through the engine as a contiguous-box read-modify-write. - - An identity `post_index` (a full-box write with no `fields`) broadcasts the - value straight into the box and writes it. Any strided/fancy/orthogonal/ - fields write instead reads the box, patches it with numpy using the same - `post_index` the read path uses (dropping the orthogonal `_Squeeze` marker - and widening the value at dropped integer axes, matching `oindex_set`), then - writes the whole box back. + fields: Fields | None, +) -> _SetSelectionPrep | None: + """Shared preamble for `_set_selection`/`_set_selection_sync`: validate + `fields`, short-circuit an empty selection (returning `None`), coerce + `value`, and decide whether the write is a full-box overwrite or a + read-modify-write. """ dtype = _native_dtype(metadata) check_fields(fields, dtype) fields = check_no_multi_fields(fields) if product(region.shape) == 0: - return + return None value_np = _coerce_write_value(value, dtype, prototype, fields) @@ -5562,25 +5752,110 @@ async def _set_selection( # those axes so it aligns with the (un-squeezed) box (matching `oindex_set`) assign_value = _widen_value_for_squeeze(value_np, sq_axes, len(region.shape)) + identity_box = None if identity_post: # full-box write: broadcast the value into the box without a read - box = np.array(np.broadcast_to(np.asarray(assign_value), region.shape), dtype=dtype) + identity_box = np.array( + np.broadcast_to(np.asarray(assign_value), region.shape), dtype=dtype + ) + return _SetSelectionPrep( + identity_box=identity_box, + fields=fields, + stripped=stripped, + assign_value=assign_value, + ) + + +def _patch_selection_box(raw: NDArrayLike, prep: _SetSelectionPrep) -> NDArrayLike: + """Patch a box read back from the engine for the read-modify-write path + (`prep.identity_box is None`), using the same `post_index` the read path + uses (dropping the orthogonal `_Squeeze` marker and widening the value at + dropped integer axes, matching `oindex_set`).""" + box = np.array(np.asarray(raw)) + target = box[prep.fields] if prep.fields else box # type: ignore[index] + if prep.stripped == (): + target[...] = prep.assign_value else: - # facade-level read-modify-write for strided/fancy/orthogonal/fields writes - box = np.array(np.asarray(await engine.read_selection(region, prototype=prototype))) - target = box[fields] if fields else box # type: ignore[index] - if stripped == (): - target[...] = assign_value - else: - target[stripped] = assign_value + target[prep.stripped] = prep.assign_value + return box - # `np.ascontiguousarray` would promote a 0-d box (scalar array) to shape (1,); - # a 0-d array is already contiguous, so pass it through unchanged. + +def _finish_set_selection(box: NDArrayLike, prototype: BufferPrototype) -> NDBuffer: + """Shared postamble for `_set_selection`/`_set_selection_sync`: make the + box contiguous (0-d arrays pass through unchanged -- `np.ascontiguousarray` + would otherwise promote them to shape `(1,)`) and wrap it for + `engine.write_selection`.""" contiguous_box = box if box.ndim == 0 else np.ascontiguousarray(box) - value_buffer = prototype.nd_buffer.from_ndarray_like(contiguous_box) + return prototype.nd_buffer.from_ndarray_like(contiguous_box) + + +async def _set_selection( + engine: AsyncArrayEngine, + metadata: ArrayMetadata, + config: ArrayConfig, + region: Region, + post_index: tuple[Any, ...], + value: npt.ArrayLike, + *, + prototype: BufferPrototype, + fields: Fields | None = None, +) -> None: + """Write a selection through the engine as a contiguous-box read-modify-write. + + An identity `post_index` (a full-box write with no `fields`) broadcasts the + value straight into the box and writes it. Any strided/fancy/orthogonal/ + fields write instead reads the box, patches it with numpy using the same + `post_index` the read path uses, then writes the whole box back. + """ + prep = _prepare_set_selection( + metadata, region, post_index, value, prototype=prototype, fields=fields + ) + if prep is None: + return + + if prep.identity_box is not None: + box = prep.identity_box + else: + # facade-level read-modify-write for strided/fancy/orthogonal/fields writes + raw = await engine.read_selection(region, prototype=prototype) + box = _patch_selection_box(raw, prep) + + value_buffer = _finish_set_selection(box, prototype) await engine.write_selection(region, value_buffer, prototype=prototype) +def _set_selection_sync( + engine: ArrayEngine, + metadata: ArrayMetadata, + config: ArrayConfig, + region: Region, + post_index: tuple[Any, ...], + value: npt.ArrayLike, + *, + prototype: BufferPrototype, + fields: Fields | None = None, +) -> None: + """Synchronous mirror of `_set_selection`: same logic, calling + `engine.read_selection`/`engine.write_selection` directly instead of + awaiting them, so `Array`'s data path never runs a coroutine on the + caller's thread. + """ + prep = _prepare_set_selection( + metadata, region, post_index, value, prototype=prototype, fields=fields + ) + if prep is None: + return + + if prep.identity_box is not None: + box = prep.identity_box + else: + raw = engine.read_selection(region, prototype=prototype) + box = _patch_selection_box(raw, prep) + + value_buffer = _finish_set_selection(box, prototype) + engine.write_selection(region, value_buffer, prototype=prototype) + + def _basic_region_post( selection: BasicSelection, shape: tuple[int, ...] ) -> tuple[Region, tuple[slice | int, ...]]: diff --git a/tests/engine/test_asyncarray_wiring.py b/tests/engine/test_asyncarray_wiring.py index 43c1530016..90fc87c2ed 100644 --- a/tests/engine/test_asyncarray_wiring.py +++ b/tests/engine/test_asyncarray_wiring.py @@ -37,9 +37,10 @@ def with_metadata(self, metadata: ArrayMetadata) -> _SpyEngine: async def test_asyncarray_routes_io_through_engine() -> None: - # NOTE: the async variant of the spy test. Task 6 gives the sync `Array` its - # own sync engine; until then the sync `z[...]` path shares this async engine, - # but this task exercises the async `AsyncArray` methods directly. + # NOTE: the async variant of the spy test. `Array` (the sync facade) now + # resolves and calls its own sync engine (see tests/engine/test_sync_path.py); + # this test exercises the async `AsyncArray` methods directly, via its + # separate async engine. z = zarr.create_array(MemoryStore(), shape=(10,), chunks=(3,), dtype="int16") aa = z.async_array spy = _SpyEngine( diff --git a/tests/engine/test_sync_path.py b/tests/engine/test_sync_path.py new file mode 100644 index 0000000000..8b074b95ef --- /dev/null +++ b/tests/engine/test_sync_path.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +import asyncio +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +import zarr +from zarr.abc.engine import ArrayEngine +from zarr.core.engine import DefaultArrayEngine +from zarr.storage import MemoryStore + +if TYPE_CHECKING: + from zarr.abc.engine import Region + from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer + from zarr.core.metadata import ArrayMetadata + + +def test_array_has_sync_engine() -> None: + z = zarr.create_array(MemoryStore(), shape=(6,), chunks=(2,), dtype="uint8") + assert isinstance(z.engine, ArrayEngine) + assert isinstance(z.engine, DefaultArrayEngine) + + +def test_array_engine_is_cached() -> None: + z = zarr.create_array(MemoryStore(), shape=(6,), chunks=(2,), dtype="uint8") + assert z.engine is z.engine + + +class _NoLoopEngine: + """A sync engine that asserts no event loop is running when called.""" + + def __init__(self, inner: ArrayEngine) -> None: + self._inner = inner + self.calls = 0 + + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + self.calls += 1 + with pytest.raises(RuntimeError): + asyncio.get_running_loop() + return self._inner.read_selection(selection, prototype=prototype) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + self.calls += 1 + with pytest.raises(RuntimeError): + asyncio.get_running_loop() + return self._inner.write_selection(selection, value, prototype=prototype) + + def with_metadata(self, metadata: ArrayMetadata) -> _NoLoopEngine: + return _NoLoopEngine(self._inner.with_metadata(metadata)) + + +def test_sync_data_path_runs_without_event_loop_in_caller_thread() -> None: + z = zarr.create_array(MemoryStore(), shape=(6,), chunks=(2,), dtype="uint8") + probe = _NoLoopEngine(z.engine) + object.__setattr__(z, "_engine", probe) # match the attribute name used in impl + + z[1:5] = np.arange(4, dtype="uint8") + out = z[1:5] + + assert probe.calls == 2 + np.testing.assert_array_equal(np.asarray(out), np.arange(4, dtype="uint8")) + + +def test_resize_rebinds_cached_sync_engine() -> None: + """After `resize`, reads/writes beyond the old bounds must go through an + engine bound to the new metadata, not a stale cached one.""" + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="uint8") + z[:] = np.arange(4, dtype="uint8") + assert np.asarray(z[:]).tolist() == [0, 1, 2, 3] + + # force engine resolution before the resize so the cache is populated + assert isinstance(z.engine, DefaultArrayEngine) + + z.resize((8,)) + z[4:8] = np.arange(4, 8, dtype="uint8") + out = z[:] + + np.testing.assert_array_equal(np.asarray(out), np.arange(8, dtype="uint8")) From 304bcbe7128740a6d91ffc7c10f9de0b544732b7 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 19:21:04 +0200 Subject: [PATCH 57/75] feat(engine): engine= parameter on array creation/open APIs Threads a public `engine=` keyword through `zarr.create_array`, `zarr.open_array`, `zarr.Group.create_array`, and their async counterparts down to where AsyncArray/Array instances are constructed. Async entry points accept an AsyncArrayEngine|name|None and pass it through unchanged; sync entry points accept ArrayEngine|AsyncArrayEngine |name|None and route it via `route_sync_engine_arg`: a name reaches both the inner AsyncArray and the outer Array so sync/async access share an engine family, a sync instance only reaches Array, and an async instance is rejected immediately (fail-fast) rather than lazily on first access. Enforces engine-kind at the two resolver choke points (resolve_async_engine/resolve_sync_engine in src/zarr/core/engine/_resolve.py) via a shared classify_engine_arg helper: a sync ArrayEngine instance handed to an AsyncArray, or an async AsyncArrayEngine instance handed to a sync Array, raises TypeError naming both what was received and what's required; an object missing read_selection entirely raises TypeError naming both protocols. Each error case gets its own test in tests/engine/test_resolve.py; tests/engine/test_engine_param.py covers the public-API threading and the fail-fast check. Assisted-by: ClaudeCode:claude-sonnet-5 --- src/zarr/api/asynchronous.py | 31 +++++++++++- src/zarr/api/synchronous.py | 40 ++++++++++++++-- src/zarr/core/array.py | 50 ++++++++++++++++--- src/zarr/core/engine/__init__.py | 4 ++ src/zarr/core/engine/_resolve.py | 74 +++++++++++++++++++++++++++- src/zarr/core/group.py | 23 ++++++++- tests/engine/test_engine_param.py | 80 +++++++++++++++++++++++++++++++ tests/engine/test_resolve.py | 60 +++++++++++++++++++++++ 8 files changed, 349 insertions(+), 13 deletions(-) create mode 100644 tests/engine/test_engine_param.py diff --git a/src/zarr/api/asynchronous.py b/src/zarr/api/asynchronous.py index 7f185535df..d563879350 100644 --- a/src/zarr/api/asynchronous.py +++ b/src/zarr/api/asynchronous.py @@ -53,9 +53,11 @@ from collections.abc import Iterable from zarr.abc.codec import Codec + from zarr.abc.engine import ArrayEngine, AsyncArrayEngine from zarr.abc.numcodec import Numcodec from zarr.core.buffer import NDArrayLikeOrScalar from zarr.core.chunk_key_encodings import ChunkKeyEncoding + from zarr.core.engine import EngineName from zarr.core.metadata.v2 import CompressorLikev2 from zarr.storage import StoreLike from zarr.types import AnyArray, AnyAsyncArray @@ -881,6 +883,7 @@ async def create( dimension_names: DimensionNamesLike = None, storage_options: dict[str, Any] | None = None, config: ArrayConfigLike | None = None, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, **kwargs: Any, ) -> AnyAsyncArray: """Create an array. @@ -1001,6 +1004,12 @@ async def create( config : ArrayConfigLike, optional Runtime configuration of the array. If provided, will override the default values from `zarr.config.array`. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the created array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. A synchronous + `ArrayEngine` instance only makes sense from the sync API; an + `AsyncArrayEngine` instance only from the async API; a name works + from either. When omitted, the `"default"` behavior is unchanged. Returns ------- @@ -1064,6 +1073,10 @@ async def create( dimension_names=dimension_names, attributes=attributes, config=config_parsed, + # `AsyncArray._create` accepts only `AsyncArrayEngine`; a wrong-kind + # (sync) instance is rejected downstream, when `AsyncArray.__init__` + # resolves it via `resolve_async_engine`. + engine=cast("AsyncArrayEngine | EngineName | None", engine), **kwargs, ) @@ -1204,6 +1217,7 @@ async def open_array( zarr_format: ZarrFormat | None = None, path: PathLike = "", storage_options: dict[str, Any] | None = None, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, **kwargs: Any, # TODO: type kwargs as valid args to save ) -> AnyAsyncArray: """Open an array using file-mode-like semantics. @@ -1221,6 +1235,13 @@ async def open_array( storage_options : dict If using an fsspec URL to create the store, these will be passed to the backend implementation. Ignored otherwise. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the opened (or, if missing, created) + array: a name (`"default"`, `"zarrista"`) or a pre-built engine + instance. A synchronous `ArrayEngine` instance only makes sense from + the sync API; an `AsyncArrayEngine` instance only from the async API; + a name works from either. When omitted, the `"default"` behavior is + unchanged. **kwargs Any keyword arguments to pass to [`create`][zarr.api.asynchronous.create]. @@ -1237,7 +1258,14 @@ async def open_array( _warn_write_empty_chunks_kwarg() try: - return await AsyncArray.open(store_path, zarr_format=zarr_format) + # `AsyncArray.open` accepts only `AsyncArrayEngine`; a wrong-kind + # (sync) instance is rejected downstream, when `AsyncArray.__init__` + # resolves it via `resolve_async_engine`. + return await AsyncArray.open( + store_path, + zarr_format=zarr_format, + engine=cast("AsyncArrayEngine | EngineName | None", engine), + ) except FileNotFoundError as err: if not store_path.read_only and mode in _CREATE_MODES: overwrite = _infer_overwrite(mode) @@ -1246,6 +1274,7 @@ async def open_array( store=store_path, zarr_format=_zarr_format, overwrite=overwrite, + engine=engine, **kwargs, ) msg = f"No array found in store {store_path.store} at path {store_path.path}" diff --git a/src/zarr/api/synchronous.py b/src/zarr/api/synchronous.py index 8386427b3f..d39db0ed4e 100644 --- a/src/zarr/api/synchronous.py +++ b/src/zarr/api/synchronous.py @@ -7,6 +7,7 @@ import zarr.api.asynchronous as async_api import zarr.core.array from zarr.core.array import DEFAULT_FILL_VALUE, Array, AsyncArray, CompressorLike +from zarr.core.engine import route_sync_engine_arg from zarr.core.group import Group from zarr.core.sync import sync from zarr.core.sync_group import create_hierarchy @@ -19,6 +20,7 @@ import numpy.typing as npt from zarr.abc.codec import Codec + from zarr.abc.engine import ArrayEngine, AsyncArrayEngine from zarr.abc.numcodec import Numcodec from zarr.api.asynchronous import ArrayLike, PathLike from zarr.core.array import ( @@ -40,6 +42,7 @@ ZarrFormat, ) from zarr.core.dtype import ZDTypeLike + from zarr.core.engine import EngineName from zarr.storage import StoreLike from zarr.types import AnyArray @@ -634,6 +637,7 @@ def create( dimension_names: DimensionNamesLike = None, storage_options: dict[str, Any] | None = None, config: ArrayConfigLike | None = None, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, **kwargs: Any, ) -> AnyArray: """Create an array. @@ -754,12 +758,19 @@ def create( config : ArrayConfigLike, optional Runtime configuration of the array. If provided, will override the default values from `zarr.config.array`. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the created array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. A synchronous + `ArrayEngine` instance only makes sense from the sync API; an + `AsyncArrayEngine` instance only from the async API; a name works + from either. When omitted, the `"default"` behavior is unchanged. Returns ------- z : Array The array. """ + engine_for_async, engine_for_array = route_sync_engine_arg(engine) return Array( sync( async_api.create( @@ -790,9 +801,11 @@ def create( dimension_names=dimension_names, storage_options=storage_options, config=config, + engine=engine_for_async, **kwargs, ) - ) + ), + engine_spec=engine_for_array, ) @@ -818,6 +831,7 @@ def create_array( overwrite: bool = False, config: ArrayConfigLike | None = None, write_data: bool = True, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, ) -> AnyArray: """Create an array. @@ -924,6 +938,12 @@ def create_array( then ``write_data`` determines whether the values in that array-like object should be written to the Zarr array created by this function. If ``write_data`` is ``False``, then the array will be left empty. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the created array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. A synchronous + `ArrayEngine` instance only makes sense from the sync API; an + `AsyncArrayEngine` instance only from the async API; a name works + from either. When omitted, the `"default"` behavior is unchanged. Returns ------- @@ -944,6 +964,7 @@ def create_array( # ``` """ + engine_for_async, engine_for_array = route_sync_engine_arg(engine) return Array( sync( zarr.core.array.create_array( @@ -967,8 +988,10 @@ def create_array( overwrite=overwrite, config=config, write_data=write_data, + engine=engine_for_async, ) - ) + ), + engine_spec=engine_for_array, ) @@ -1330,6 +1353,7 @@ def open_array( zarr_format: ZarrFormat | None = None, path: PathLike = "", storage_options: dict[str, Any] | None = None, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, **kwargs: Any, ) -> AnyArray: """Open an array using file-mode-like semantics. @@ -1347,6 +1371,13 @@ def open_array( storage_options : dict If using an fsspec URL to create the store, these will be passed to the backend implementation. Ignored otherwise. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the opened (or, if missing, created) + array: a name (`"default"`, `"zarrista"`) or a pre-built engine + instance. A synchronous `ArrayEngine` instance only makes sense from + the sync API; an `AsyncArrayEngine` instance only from the async API; + a name works from either. When omitted, the `"default"` behavior is + unchanged. **kwargs Any keyword arguments to pass to [`create`][zarr.api.asynchronous.create]. @@ -1356,6 +1387,7 @@ def open_array( AsyncArray The opened array. """ + engine_for_async, engine_for_array = route_sync_engine_arg(engine) return Array( sync( async_api.open_array( @@ -1363,9 +1395,11 @@ def open_array( zarr_format=zarr_format, path=path, storage_options=storage_options, + engine=engine_for_async, **kwargs, ) - ) + ), + engine_spec=engine_for_array, ) diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 63e46ef141..33868d34cc 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -90,6 +90,7 @@ normalize_orthogonal, resolve_async_engine, resolve_sync_engine, + route_sync_engine_arg, squeeze_axes, strip_squeeze, ) @@ -433,6 +434,7 @@ async def _create( overwrite: bool = False, data: npt.ArrayLike | None = None, config: ArrayConfigLike | None = None, + engine: AsyncArrayEngine | EngineName | None = None, ) -> AnyAsyncArray: """Method to create a new asynchronous array instance. Deprecated in favor of [`zarr.api.asynchronous.create_array`][]. @@ -489,6 +491,7 @@ async def _create( overwrite=overwrite, config=config_parsed, chunk_grid=chunk_grid, + engine=engine, ) elif zarr_format == 2: if codecs is not None: @@ -533,6 +536,7 @@ async def _create( compressor=compressor, attributes=attributes, overwrite=overwrite, + engine=engine, ) else: raise ValueError(f"zarr_format must be 2 or 3, got {zarr_format}") # pragma: no cover @@ -612,6 +616,7 @@ async def _create_v3( dimension_names: DimensionNamesLike = None, attributes: dict[str, JSON] | None = None, overwrite: bool = False, + engine: AsyncArrayEngine | EngineName | None = None, ) -> AsyncArrayV3: if overwrite: if store_path.store.supports_deletes: @@ -639,7 +644,7 @@ async def _create_v3( attributes=attributes, ) - array = cls(metadata=metadata, store_path=store_path, config=config) + array = cls(metadata=metadata, store_path=store_path, config=config, engine=engine) await array._save_metadata(metadata, ensure_parents=True) return array @@ -693,6 +698,7 @@ async def _create_v2( compressor: CompressorLike = "auto", attributes: dict[str, JSON] | None = None, overwrite: bool = False, + engine: AsyncArrayEngine | EngineName | None = None, ) -> AsyncArrayV2: if overwrite: if store_path.store.supports_deletes: @@ -728,7 +734,7 @@ async def _create_v2( attributes=attributes, ) - array = cls(metadata=metadata, store_path=store_path, config=config) + array = cls(metadata=metadata, store_path=store_path, config=config, engine=engine) await array._save_metadata(metadata, ensure_parents=True) return array @@ -769,6 +775,7 @@ async def open( cls, store: StoreLike, zarr_format: ZarrFormat | None = 3, + engine: AsyncArrayEngine | EngineName | None = None, ) -> AnyAsyncArray: """ Async method to open an existing Zarr array from a given store. @@ -781,6 +788,10 @@ async def open( for a description of all valid StoreLike values. zarr_format : ZarrFormat | None, optional The Zarr format version (default is 3). + engine : AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the opened array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. When omitted, the + `"default"` engine is used. Returns ------- @@ -812,7 +823,7 @@ async def example(): metadata_dict = await get_array_metadata(store_path, zarr_format=zarr_format) # TODO: remove this cast when we have better type hints _metadata_dict = cast("ArrayMetadataJSON_V3", metadata_dict) - return cls(store_path=store_path, metadata=_metadata_dict) + return cls(store_path=store_path, metadata=_metadata_dict, engine=engine) @property def store(self) -> Store: @@ -1920,10 +1931,12 @@ def _create( # runtime overwrite: bool = False, config: ArrayConfigLike | None = None, + engine: ArrayEngine | EngineName | None = None, ) -> Self: """Creates a new Array instance from an initialized store. Deprecated in favor of [`zarr.create_array`][]. """ + engine_for_async, engine_for_array = route_sync_engine_arg(engine) async_array = sync( AsyncArray._create( store=store, @@ -1943,9 +1956,10 @@ def _create( compressor=compressor, overwrite=overwrite, config=config, + engine=engine_for_async, ), ) - return cls(async_array) + return cls(async_array, engine_spec=engine_for_array) @classmethod def from_dict( @@ -1982,6 +1996,7 @@ def from_dict( def open( cls, store: StoreLike, + engine: ArrayEngine | EngineName | None = None, ) -> Self: """Opens an existing Array from a store. @@ -1991,14 +2006,19 @@ def open( Store containing the Array. See the [storage documentation in the user guide][user-guide-store-like] for a description of all valid StoreLike values. + engine : ArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the opened array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. When omitted, the + `"default"` behavior is unchanged. Returns ------- Array Array opened from the store. """ - async_array = sync(AsyncArray.open(store)) - return cls(async_array) + engine_for_async, engine_for_array = route_sync_engine_arg(engine) + async_array = sync(AsyncArray.open(store, engine=engine_for_async)) + return cls(async_array, engine_spec=engine_for_array) @property def store(self) -> Store: @@ -4163,6 +4183,7 @@ async def from_array( storage_options: dict[str, Any] | None = None, overwrite: bool = False, config: ArrayConfigLike | None = None, + engine: AsyncArrayEngine | EngineName | None = None, ) -> AnyAsyncArray: """Create an array from an existing array or array-like. @@ -4387,6 +4408,7 @@ async def from_array( dimension_names=dimension_names, overwrite=overwrite, config=config_parsed, + engine=engine, ) if write_data: @@ -4436,6 +4458,7 @@ async def init_array( dimension_names: DimensionNamesLike = None, overwrite: bool = False, config: ArrayConfigLike | None = None, + engine: AsyncArrayEngine | EngineName | None = None, ) -> AnyAsyncArray: """Create and persist an array metadata document. @@ -4641,7 +4664,7 @@ async def init_array( attributes=attributes, ) - arr = AsyncArray(metadata=meta, store_path=store_path, config=config) + arr = AsyncArray(metadata=meta, store_path=store_path, config=config, engine=engine) await arr._save_metadata(meta, ensure_parents=True) return arr @@ -4668,6 +4691,7 @@ async def create_array( overwrite: bool = False, config: ArrayConfigLike | None = None, write_data: bool = True, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, ) -> AnyAsyncArray: """Create an array. @@ -4772,6 +4796,12 @@ async def create_array( then ``write_data`` determines whether the values in that array-like object should be written to the Zarr array created by this function. If ``write_data`` is ``False``, then the array will be left empty. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the created array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. A synchronous + `ArrayEngine` instance only makes sense from the sync API; an + `AsyncArrayEngine` instance only from the async API; a name works + from either. When omitted, the `"default"` behavior is unchanged. Returns ------- @@ -4816,6 +4846,10 @@ async def create_array( storage_options=storage_options, overwrite=overwrite, config=config, + # `from_array` accepts only `AsyncArrayEngine`; a wrong-kind (sync) + # instance is rejected downstream, when `AsyncArray.__init__` + # resolves it via `resolve_async_engine`. + engine=cast("AsyncArrayEngine | EngineName | None", engine), ) else: mode: Literal["a"] = "a" @@ -4840,6 +4874,8 @@ async def create_array( dimension_names=dimension_names, overwrite=overwrite, config=config, + # see the comment above the analogous `from_array` call. + engine=cast("AsyncArrayEngine | EngineName | None", engine), ) diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py index 2229111d48..0d99cf4f0b 100644 --- a/src/zarr/core/engine/__init__.py +++ b/src/zarr/core/engine/__init__.py @@ -15,8 +15,10 @@ ) from zarr.core.engine._resolve import ( EngineName, + classify_engine_arg, resolve_async_engine, resolve_sync_engine, + route_sync_engine_arg, ) __all__ = [ @@ -26,12 +28,14 @@ "DefaultHierarchyEngine", "EngineName", "apply_post_index", + "classify_engine_arg", "normalize_basic", "normalize_block", "normalize_coordinate", "normalize_orthogonal", "resolve_async_engine", "resolve_sync_engine", + "route_sync_engine_arg", "squeeze_axes", "strip_squeeze", ] diff --git a/src/zarr/core/engine/_resolve.py b/src/zarr/core/engine/_resolve.py index f831f5aed7..085a6ed41f 100644 --- a/src/zarr/core/engine/_resolve.py +++ b/src/zarr/core/engine/_resolve.py @@ -3,6 +3,7 @@ from __future__ import annotations import contextlib +import inspect import weakref from typing import TYPE_CHECKING, Literal @@ -24,10 +25,71 @@ from zarr.core.array_spec import ArrayConfig from zarr.core.metadata import ArrayMetadata -__all__ = ["EngineName", "resolve_async_engine", "resolve_sync_engine"] +__all__ = [ + "EngineName", + "classify_engine_arg", + "resolve_async_engine", + "resolve_sync_engine", + "route_sync_engine_arg", +] EngineName = Literal["default", "zarrista"] + +def classify_engine_arg(engine: object) -> Literal["name", "sync", "async"]: + """Classify an `engine=` argument as a name, a sync instance, or an async instance. + + `None` and `str` values classify as `"name"` -- valid wherever an `engine=` + argument is accepted. Any other value must implement `read_selection`: a + coroutine function classifies as `"async"` (an `AsyncArrayEngine`), anything + else as `"sync"` (an `ArrayEngine`). Objects with no `read_selection` at all + raise `TypeError` naming the two protocols. + """ + if engine is None or isinstance(engine, str): + return "name" + read_selection = getattr(engine, "read_selection", None) + if read_selection is None: + raise TypeError( + f"{engine!r} does not implement the ArrayEngine or AsyncArrayEngine protocol " + "(missing a `read_selection` method)" + ) + return "async" if inspect.iscoroutinefunction(read_selection) else "sync" + + +def route_sync_engine_arg( + engine: ArrayEngine | AsyncArrayEngine | EngineName | None, +) -> tuple[AsyncArrayEngine | EngineName | None, ArrayEngine | EngineName | None]: + """Route a sync-entry-point `engine=` argument to its two consumers. + + The public sync entry points (`zarr.create_array`, `zarr.open_array`, ...) + accept the same broad `engine` type as their async counterparts so their + signatures and docstrings match; this function is where the sync-specific + rules actually get enforced. + + Returns `(engine_for_async_array, engine_for_array)`. A name (or `None`) is + valid for both layers and is returned unchanged in both slots, so sync and + async access to the same object use the same engine family. A sync + `ArrayEngine` instance is returned only in the second slot -- the wrapped + `AsyncArray` keeps its default engine. An `AsyncArrayEngine` instance cannot + serve a sync entry point and raises `TypeError`. + """ + kind = classify_engine_arg(engine) + if kind == "async": + # Fail fast at the API boundary rather than lazily when `Array` + # resolves its engine; message kept identical to + # `resolve_sync_engine`'s so the error looks the same regardless of + # where the wrong-kind instance was actually caught. + raise TypeError( + "Array requires a synchronous engine (ArrayEngine); got an " + f"async engine of type `{type(engine).__name__}`" + ) + if kind == "name": + return engine, engine # type: ignore[return-value] + # kind == "sync": the instance only serves the sync Array; the inner + # AsyncArray keeps its default engine. + return None, engine # type: ignore[return-value] + + # (name, kind, id(store)) -> hierarchy engine; entries evicted automatically once # nothing keeps the hierarchy engine itself alive (see `_keepalive` below). # @@ -114,6 +176,11 @@ def resolve_async_engine( engine, "async", store, factory ) return _keepalive(hierarchy.array_engine(path, metadata, config), hierarchy) # type: ignore[return-value] + if classify_engine_arg(engine) == "sync": + raise TypeError( + "AsyncArray requires an async engine (AsyncArrayEngine); got a " + f"synchronous engine of type `{type(engine).__name__}`" + ) return engine @@ -134,4 +201,9 @@ def resolve_sync_engine( engine, "sync", store, factory ) return _keepalive(hierarchy.array_engine(path, metadata, config), hierarchy) # type: ignore[return-value] + if classify_engine_arg(engine) == "async": + raise TypeError( + "Array requires a synchronous engine (ArrayEngine); got an " + f"async engine of type `{type(engine).__name__}`" + ) return engine diff --git a/src/zarr/core/group.py b/src/zarr/core/group.py index 52eaa3e144..c75a86f2a4 100644 --- a/src/zarr/core/group.py +++ b/src/zarr/core/group.py @@ -47,6 +47,7 @@ parse_shapelike, ) from zarr.core.config import config +from zarr.core.engine import route_sync_engine_arg from zarr.core.metadata import ArrayV2Metadata, ArrayV3Metadata from zarr.core.metadata.io import save_metadata from zarr.core.sync import SyncMixin, sync @@ -73,11 +74,13 @@ ) from typing import Any + from zarr.abc.engine import ArrayEngine, AsyncArrayEngine from zarr.core.array_spec import ArrayConfigLike from zarr.core.buffer import Buffer, BufferPrototype from zarr.core.chunk_key_encodings import ChunkKeyEncodingLike from zarr.core.common import MemoryOrder from zarr.core.dtype import ZDTypeLike + from zarr.core.engine import EngineName from zarr.types import AnyArray, AnyAsyncArray, ArrayV2, ArrayV3, AsyncArrayV2, AsyncArrayV3 logger = logging.getLogger("zarr.group") @@ -1029,6 +1032,7 @@ async def create_array( overwrite: bool = False, config: ArrayConfigLike | None = None, write_data: bool = True, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, ) -> AnyAsyncArray: """Create an array within this group. @@ -1122,6 +1126,12 @@ async def create_array( then ``write_data`` determines whether the values in that array-like object should be written to the Zarr array created by this function. If ``write_data`` is ``False``, then the array will be left empty. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the created array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. A synchronous + `ArrayEngine` instance only makes sense from the sync API; an + `AsyncArrayEngine` instance only from the async API; a name works + from either. When omitted, the `"default"` behavior is unchanged. Returns ------- @@ -1152,6 +1162,7 @@ async def create_array( overwrite=overwrite, config=config, write_data=write_data, + engine=engine, ) async def require_array( @@ -2543,6 +2554,7 @@ def create_array( overwrite: bool = False, config: ArrayConfigLike | None = None, write_data: bool = True, + engine: ArrayEngine | AsyncArrayEngine | EngineName | None = None, ) -> AnyArray: """Create an array within this group. @@ -2638,6 +2650,12 @@ def create_array( then ``write_data`` determines whether the values in that array-like object should be written to the Zarr array created by this function. If ``write_data`` is ``False``, then the array will be left empty. + engine : ArrayEngine | AsyncArrayEngine | Literal["default", "zarrista"] | None, optional + The data-path engine backing the created array: a name (`"default"`, + `"zarrista"`) or a pre-built engine instance. A synchronous + `ArrayEngine` instance only makes sense from the sync API; an + `AsyncArrayEngine` instance only from the async API; a name works + from either. When omitted, the `"default"` behavior is unchanged. Returns ------- @@ -2646,6 +2664,7 @@ def create_array( compressors = _parse_deprecated_compressor( compressor, compressors, zarr_format=self.metadata.zarr_format ) + engine_for_async, engine_for_array = route_sync_engine_arg(engine) return Array( self._sync( self._async_group.create_array( @@ -2667,8 +2686,10 @@ def create_array( storage_options=storage_options, config=config, write_data=write_data, + engine=engine_for_async, ) - ) + ), + engine_spec=engine_for_array, ) def require_array(self, name: str, *, shape: ShapeLike, **kwargs: Any) -> AnyArray: diff --git a/tests/engine/test_engine_param.py b/tests/engine/test_engine_param.py new file mode 100644 index 0000000000..5c130c6e8c --- /dev/null +++ b/tests/engine/test_engine_param.py @@ -0,0 +1,80 @@ +"""`engine=` threading through `zarr.create_array` / `zarr.open_array` and their +async counterparts (Task 7 of the array-engine-protocol plan).""" + +from __future__ import annotations + +import numpy as np +import pytest + +import zarr +from zarr.core.array import AsyncArray +from zarr.core.engine import DefaultArrayEngine, DefaultAsyncArrayEngine +from zarr.storage import MemoryStore + + +def test_engine_param_combinations() -> None: + store = MemoryStore() + z = zarr.create_array(store, name="a", shape=(4,), chunks=(2,), dtype="int8", engine="default") + z[:] = np.arange(4, dtype="int8") + assert isinstance(z.engine, DefaultArrayEngine) + + z2 = zarr.open_array(store, path="a", engine="default") + np.testing.assert_array_equal(np.asarray(z2[:]), np.arange(4, dtype="int8")) + assert isinstance(z2.async_array.engine, DefaultAsyncArrayEngine) + + # user-provided sync instance + inst = z2.engine + z3 = zarr.open_array(store, path="a", engine=inst) + assert z3.engine is inst + # the wrapped AsyncArray keeps its own (default) engine -- a sync instance + # must never reach AsyncArray. + assert isinstance(z3.async_array.engine, DefaultAsyncArrayEngine) + + +def test_engine_param_unknown_name() -> None: + with pytest.raises(ValueError, match="unknown engine"): + zarr.create_array( + MemoryStore(), + shape=(2,), + chunks=(2,), + dtype="int8", + engine="nope", # type: ignore[arg-type] + ) + + +def test_async_array_rejects_sync_engine_instance() -> None: + """A sync `ArrayEngine` instance passed to `AsyncArray` -- any construction + path, not just the public API -- must raise `TypeError` naming both + protocols.""" + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + sync_engine = z.engine + aa = z.async_array + with pytest.raises(TypeError, match="ArrayEngine"): + AsyncArray( + metadata=aa.metadata, + store_path=aa.store_path, + engine=sync_engine, # type: ignore[call-overload] + ) + + +def test_sync_api_rejects_async_engine_instance() -> None: + """An `AsyncArrayEngine` instance passed to the sync `create_array` entry + point must raise `TypeError` immediately (not lazily on first data + access).""" + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + async_engine = z.async_array.engine + with pytest.raises(TypeError, match="ArrayEngine"): + zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8", engine=async_engine) + + +def test_engine_missing_read_selection_raises_type_error() -> None: + """An object implementing neither engine protocol (no `read_selection`) + must raise `TypeError` naming the protocols, not some other error.""" + with pytest.raises(TypeError, match="read_selection"): + zarr.create_array( + MemoryStore(), + shape=(4,), + chunks=(2,), + dtype="int8", + engine=object(), # type: ignore[arg-type] + ) diff --git a/tests/engine/test_resolve.py b/tests/engine/test_resolve.py index 32bddc5fab..73f09dc8ae 100644 --- a/tests/engine/test_resolve.py +++ b/tests/engine/test_resolve.py @@ -99,3 +99,63 @@ def test_zarrista_missing_raises_import_error() -> None: resolve_async_engine( "zarrista", store=z.store, path=z.path, metadata=z.async_array.metadata ) + + +def test_resolve_async_engine_rejects_sync_engine_instance() -> None: + """`resolve_async_engine` must reject a synchronous `ArrayEngine` instance -- + `AsyncArray` can only be backed by an `AsyncArrayEngine`.""" + z = _array() + sync_engine = resolve_sync_engine( + None, store=z.store, path=z.path, metadata=z.async_array.metadata + ) + with pytest.raises(TypeError, match="AsyncArray"): + resolve_async_engine( + sync_engine, # type: ignore[arg-type] + store=z.store, + path=z.path, + metadata=z.async_array.metadata, + ) + + +def test_resolve_sync_engine_rejects_async_engine_instance() -> None: + """`resolve_sync_engine` must reject an asynchronous `AsyncArrayEngine` + instance -- `Array` can only be backed by a synchronous `ArrayEngine`.""" + z = _array() + async_engine = resolve_async_engine( + None, store=z.store, path=z.path, metadata=z.async_array.metadata + ) + with pytest.raises(TypeError, match="Array"): + resolve_sync_engine( + async_engine, # type: ignore[arg-type] + store=z.store, + path=z.path, + metadata=z.async_array.metadata, + ) + + +def test_resolve_async_engine_rejects_object_missing_read_selection() -> None: + """An object implementing neither engine protocol (no `read_selection`) must + raise `TypeError` naming the protocols, not some other error, when handed to + `resolve_async_engine`.""" + z = _array() + with pytest.raises(TypeError, match="read_selection"): + resolve_async_engine( + object(), # type: ignore[arg-type] + store=z.store, + path=z.path, + metadata=z.async_array.metadata, + ) + + +def test_resolve_sync_engine_rejects_object_missing_read_selection() -> None: + """An object implementing neither engine protocol (no `read_selection`) must + raise `TypeError` naming the protocols, not some other error, when handed to + `resolve_sync_engine`.""" + z = _array() + with pytest.raises(TypeError, match="read_selection"): + resolve_sync_engine( + object(), # type: ignore[arg-type] + store=z.store, + path=z.path, + metadata=z.async_array.metadata, + ) From 76016953358b898c896fe867ac7ddaaf3fbe94ca Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 20:45:39 +0200 Subject: [PATCH 58/75] feat: zarrista-backed array engines with store translation Adds `zarr.zarrista`: `ZarristaHierarchyEngine`/`ZarristaEngine` (sync) and `ZarristaAsyncHierarchyEngine`/`ZarristaAsyncEngine` (async), backing `engine="zarrista"` resolution with the Rust `zarrs`-based zarrista package. - `_translate.py` maps zarr stores to zarrista stores: `LocalStore` -> `zarrista.store.FilesystemStore` (sync); obstore-backed `ObjectStore` or an icechunk store -> the underlying obstore/icechunk instance (async). - `_engine.py` reads through `retrieve_array_subset` and writes via the engine's own chunk decomposition (RMW through `retrieve_chunk`/ `store_chunk`), correctly falling back to RMW on edge chunks whose clipped extent doesn't match the nominal chunk shape, and rejecting non-`Tensor` decoded layouts (vlen/masked) it cannot map to a zarr-python array. `ZarristaAsyncEngine`/`ZarristaAsyncHierarchyEngine` defer store translation to first I/O rather than construction time, since `AsyncArray.__init__` always eagerly resolves an async engine even when only the sync `Array` is ever used. - Adds the `zarrista` dependency group (git-pinned to zarrista main), plus `icechunk`/`obstore` which zarrista's `store` submodule imports unconditionally despite not declaring them as runtime dependencies. Assisted-by: ClaudeCode:claude-sonnet-5 --- pyproject.toml | 9 ++ src/zarr/core/engine/_resolve.py | 4 +- src/zarr/zarrista/__init__.py | 19 +++ src/zarr/zarrista/_engine.py | 254 +++++++++++++++++++++++++++++++ src/zarr/zarrista/_translate.py | 46 ++++++ tests/zarrista/__init__.py | 0 tests/zarrista/test_engine.py | 101 ++++++++++++ tests/zarrista/test_translate.py | 41 +++++ uv.lock | 143 ++++++++++++----- 9 files changed, 572 insertions(+), 45 deletions(-) create mode 100644 src/zarr/zarrista/__init__.py create mode 100644 src/zarr/zarrista/_engine.py create mode 100644 src/zarr/zarrista/_translate.py create mode 100644 tests/zarrista/__init__.py create mode 100644 tests/zarrista/test_engine.py create mode 100644 tests/zarrista/test_translate.py diff --git a/pyproject.toml b/pyproject.toml index 89e05068ea..932387186a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -139,6 +139,15 @@ zarrs = [ {include-group = "test"}, "zarrs-bindings", ] +zarrista = [ + {include-group = "test"}, + "zarrista @ git+https://github.com/developmentseed/zarrista@95e47ad4c414c5920f0cf15550f923039641da8e", + # zarrista's `store` submodule unconditionally imports both of these + # (for its `AsyncStore` type alias), even though neither is declared in + # zarrista's own package metadata as a runtime dependency. + "icechunk>=1.1.21", + "obstore>=0.10.1", +] [tool.coverage.report] exclude_also = [ diff --git a/src/zarr/core/engine/_resolve.py b/src/zarr/core/engine/_resolve.py index 085a6ed41f..3330e0ccd7 100644 --- a/src/zarr/core/engine/_resolve.py +++ b/src/zarr/core/engine/_resolve.py @@ -144,9 +144,7 @@ def _hierarchy_factory(name: str, *, sync: bool) -> Callable[[Store], object]: "engine='zarrista' requires the `zarrista` package; " "install zarr with the `zarrista` extra" ) from e - return ( # type: ignore[no-any-return] - ZarristaHierarchyEngine if sync else ZarristaAsyncHierarchyEngine - ) + return ZarristaHierarchyEngine if sync else ZarristaAsyncHierarchyEngine raise ValueError(f"unknown engine name {name!r}; expected 'default' or 'zarrista'") diff --git a/src/zarr/zarrista/__init__.py b/src/zarr/zarrista/__init__.py new file mode 100644 index 0000000000..4873414bd6 --- /dev/null +++ b/src/zarr/zarrista/__init__.py @@ -0,0 +1,19 @@ +"""Zarrista-backed array engines for zarr-python. + +Requires the `zarrista` package (`pip install zarr[zarrista]` once released; +currently the git-pinned `zarrista` dependency group). +""" + +from zarr.zarrista._engine import ( + ZarristaAsyncEngine, + ZarristaAsyncHierarchyEngine, + ZarristaEngine, + ZarristaHierarchyEngine, +) + +__all__ = [ + "ZarristaAsyncEngine", + "ZarristaAsyncHierarchyEngine", + "ZarristaEngine", + "ZarristaHierarchyEngine", +] diff --git a/src/zarr/zarrista/_engine.py b/src/zarr/zarrista/_engine.py new file mode 100644 index 0000000000..51aa06f7c3 --- /dev/null +++ b/src/zarr/zarrista/_engine.py @@ -0,0 +1,254 @@ +"""Zarrista-backed array engines.""" + +from __future__ import annotations + +import itertools +from typing import TYPE_CHECKING, Any, cast + +import numpy as np + +from zarr.errors import UnsupportedEngineError +from zarr.zarrista._translate import translate_store_async, translate_store_sync + +if TYPE_CHECKING: + from zarr_metadata import ArrayMetadataV3 + + from zarr.abc.engine import Region + from zarr.abc.store import Store + from zarr.core.array_spec import ArrayConfig + from zarr.core.buffer import BufferPrototype, NDBuffer + from zarr.core.metadata import ArrayMetadata + +__all__ = [ + "ZarristaAsyncEngine", + "ZarristaAsyncHierarchyEngine", + "ZarristaEngine", + "ZarristaHierarchyEngine", +] + + +def _require_v3(metadata: ArrayMetadata) -> ArrayMetadataV3: + if metadata.zarr_format != 3: + raise UnsupportedEngineError( + "the zarrista engine supports Zarr v3 only; this array is " + f"format v{metadata.zarr_format}" + ) + # `metadata.to_dict()` is a plain `dict[str, JSON]`; zarrista's stubs type + # `from_metadata`'s argument as the `ArrayMetadataV3` TypedDict, which the + # dict's runtime shape matches by construction. + return cast("ArrayMetadataV3", metadata.to_dict()) + + +def _region_to_selection(region: Region) -> tuple[slice, ...]: + return tuple(slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True)) + + +def _decoded_to_numpy(decoded: Any) -> np.ndarray[Any, Any]: + # Only `Tensor` (fixed-width, unmasked) implements `__array__`/the buffer + # protocol in a way that produces a correct zarr-python array. `VariableArray` + # (vlen data) only exposes the Arrow C Data interface, so `np.asarray` on it + # does *not* raise -- it silently wraps the opaque object in a 0-d `object` + # array -- which is why this is a type-name check rather than a try/except + # around `np.asarray`. `MaskedTensor`/`MaskedVariableArray` do convert (the + # former to a `numpy.ma.MaskedArray`), but masked layouts have no + # zarr-python equivalent to hold that result. + type_name = type(decoded).__name__ + if type_name != "Tensor": + raise NotImplementedError( + f"zarrista returned a {type_name}; only fixed-width, unmasked reads " + "(zarrista `Tensor`) are supported by this engine. vlen reads " + "(`VariableArray`) are pending zarrista numpy export; masked layouts " + "(`MaskedTensor`/`MaskedVariableArray`) have no zarr-python equivalent." + ) + return np.asarray(decoded) + + +def _chunks_overlapping( + region: Region, chunk_shape: tuple[int, ...] +) -> itertools.product[tuple[int, ...]]: + ranges = [ + range(s // c, (e + c - 1) // c) if e > s else range(0) + for s, e, c in zip(region.start, region.end_exclusive, chunk_shape, strict=True) + ] + return itertools.product(*ranges) + + +class ZarristaEngine: + """Sync engine over `zarrista.Array`. No event loop involved.""" + + def __init__(self, zarrista_array: Any) -> None: + self._arr = zarrista_array + + def with_metadata(self, metadata: ArrayMetadata) -> ZarristaEngine: + import zarrista + + return ZarristaEngine( + zarrista.Array.from_metadata(_require_v3(metadata), self._arr.store, self._arr.path) + ) + + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> Any: + return _decoded_to_numpy(self._arr.retrieve_array_subset(_region_to_selection(selection))) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + import zarrista + + value_np = np.ascontiguousarray(value.as_ndarray_like()) + # regular grids: the origin chunk has the full (unclipped) chunk shape + chunk_shape = tuple(self._arr.chunk_shape([0] * len(selection.start))) + for chunk_idx_tuple in _chunks_overlapping(selection, chunk_shape): + chunk_idx = list(chunk_idx_tuple) + chunk_slices = self._arr.chunk_subset(chunk_idx) + # overlap of the write region with this chunk, in array coords + lo = tuple( + max(cs.start, s) for cs, s in zip(chunk_slices, selection.start, strict=True) + ) + hi = tuple( + min(cs.stop, e) for cs, e in zip(chunk_slices, selection.end_exclusive, strict=True) + ) + in_value = tuple( + slice(a - s, b - s) for a, b, s in zip(lo, hi, selection.start, strict=True) + ) + in_chunk = tuple( + slice(a - cs.start, b - cs.start) + for a, b, cs in zip(lo, hi, chunk_slices, strict=True) + ) + full_chunk = all( + sl.start == 0 and sl.stop == (cs.stop - cs.start) and (cs.stop - cs.start) == c + for sl, cs, c in zip(in_chunk, chunk_slices, chunk_shape, strict=True) + ) + if full_chunk: + chunk_np = np.ascontiguousarray(value_np[in_value]) + else: + chunk_np = np.array(_decoded_to_numpy(self._arr.retrieve_chunk(chunk_idx))) + chunk_np[in_chunk] = value_np[in_value] + chunk_np = np.ascontiguousarray(chunk_np) + self._arr.store_chunk( + chunk_idx, zarrista.ArrayBytes(chunk_np.reshape(-1).view(np.uint8)) + ) + + +class ZarristaHierarchyEngine: + """Store-bound factory for sync zarrista engines (translates the store once).""" + + def __init__(self, store: Store) -> None: + self._zarr_store = store + self._zstore = translate_store_sync(store) + + def array_engine( + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None + ) -> ZarristaEngine: + """Mint a sync array engine bound to `path`/`metadata`. + + `config` is accepted for protocol conformance with `HierarchyEngine` + but is unused: zarrista owns its own codec options and does not read + zarr-python's `ArrayConfig` (e.g. `order`, `read_missing_chunks`). + """ + import zarrista + + return ZarristaEngine( + zarrista.Array.from_metadata(_require_v3(metadata), self._zstore, "/" + path.strip("/")) + ) + + +class ZarristaAsyncEngine: + """Async engine over `zarrista.AsyncArray`. + + Store translation, and construction of the underlying `zarrista.AsyncArray`, + are deferred to the first `read_selection`/`write_selection` call rather + than done at construction time. `AsyncArray.__init__` always eagerly + resolves *an* async engine -- even for a plain sync `Array` that never + touches it, since only the sync engine is lazily resolved there (see + `Array.engine`). Without this deferral, `engine="zarrista"` over a + sync-only store (e.g. `LocalStore`) would raise `UnsupportedEngineError` + just from opening the array, even when only ever accessed synchronously. + """ + + def __init__(self, store: Store, path: str, metadata: ArrayMetadata) -> None: + self._store = store + self._path = path + self._metadata = metadata + self._arr: Any | None = None + + def with_metadata(self, metadata: ArrayMetadata) -> ZarristaAsyncEngine: + _require_v3(metadata) + return ZarristaAsyncEngine(self._store, self._path, metadata) + + async def _ensure_arr(self) -> Any: + if self._arr is None: + import zarrista + + meta_dict = _require_v3(self._metadata) + zstore = translate_store_async(self._store) + self._arr = zarrista.AsyncArray.from_metadata(meta_dict, zstore, self._path) + return self._arr + + async def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> Any: + arr = await self._ensure_arr() + return _decoded_to_numpy(await arr.retrieve_array_subset(_region_to_selection(selection))) + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + import zarrista + + arr = await self._ensure_arr() + value_np = np.ascontiguousarray(value.as_ndarray_like()) + # regular grids: the origin chunk has the full (unclipped) chunk shape + chunk_shape = tuple(arr.chunk_shape([0] * len(selection.start))) + for chunk_idx_tuple in _chunks_overlapping(selection, chunk_shape): + chunk_idx = list(chunk_idx_tuple) + chunk_slices = arr.chunk_subset(chunk_idx) + lo = tuple( + max(cs.start, s) for cs, s in zip(chunk_slices, selection.start, strict=True) + ) + hi = tuple( + min(cs.stop, e) for cs, e in zip(chunk_slices, selection.end_exclusive, strict=True) + ) + in_value = tuple( + slice(a - s, b - s) for a, b, s in zip(lo, hi, selection.start, strict=True) + ) + in_chunk = tuple( + slice(a - cs.start, b - cs.start) + for a, b, cs in zip(lo, hi, chunk_slices, strict=True) + ) + full_chunk = all( + sl.start == 0 and sl.stop == (cs.stop - cs.start) and (cs.stop - cs.start) == c + for sl, cs, c in zip(in_chunk, chunk_slices, chunk_shape, strict=True) + ) + if full_chunk: + chunk_np = np.ascontiguousarray(value_np[in_value]) + else: + chunk_np = np.array(_decoded_to_numpy(await arr.retrieve_chunk(chunk_idx))) + chunk_np[in_chunk] = value_np[in_value] + chunk_np = np.ascontiguousarray(chunk_np) + await arr.store_chunk( + chunk_idx, zarrista.ArrayBytes(chunk_np.reshape(-1).view(np.uint8)) + ) + + +class ZarristaAsyncHierarchyEngine: + """Store-bound factory for async zarrista engines. + + Unlike `ZarristaHierarchyEngine`, this does *not* translate the store at + construction time -- see `ZarristaAsyncEngine` for why. + """ + + def __init__(self, store: Store) -> None: + self._zarr_store = store + + def array_engine( + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None + ) -> ZarristaAsyncEngine: + """Mint an async array engine bound to `path`/`metadata`. + + `config` is accepted for protocol conformance with + `AsyncHierarchyEngine` but is unused: zarrista owns its own codec + options and does not read zarr-python's `ArrayConfig` (e.g. `order`, + `read_missing_chunks`). `metadata` is validated as Zarr v3 eagerly + (cheap, and lets an unsupported-format error surface immediately); + the store itself is only translated lazily, on first I/O. + """ + _require_v3(metadata) + return ZarristaAsyncEngine(self._zarr_store, "/" + path.strip("/"), metadata) diff --git a/src/zarr/zarrista/_translate.py b/src/zarr/zarrista/_translate.py new file mode 100644 index 0000000000..f1bcfe1c4e --- /dev/null +++ b/src/zarr/zarrista/_translate.py @@ -0,0 +1,46 @@ +"""Translate zarr-python stores into stores Zarrista can consume.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +from zarr.errors import UnsupportedEngineError +from zarr.storage import LocalStore + +if TYPE_CHECKING: + from zarr.abc.store import Store + +_SYNC_SUPPORTED = "LocalStore" +_ASYNC_SUPPORTED = "zarr.storage.ObjectStore (obstore-backed) or an icechunk store" + + +def translate_store_sync(store: Store) -> Any: + """zarr store -> zarrista sync store (`zarrista.store.FilesystemStore`).""" + import zarrista + + if isinstance(store, LocalStore): + return zarrista.store.FilesystemStore(store.root) + raise UnsupportedEngineError( + f"the zarrista sync engine cannot serve a {type(store).__name__}; " + f"supported: {_SYNC_SUPPORTED}. Note: zarr's MemoryStore lives in the " + "Python process and cannot be shared with the Rust extension." + ) + + +def translate_store_async(store: Store) -> Any: + """zarr store -> zarrista async store (obstore `ObjectStore` or icechunk `Session`).""" + from zarr.storage import ObjectStore + + if isinstance(store, ObjectStore): + return store.store # the underlying obstore instance + try: + from icechunk import IcechunkStore + + if isinstance(store, IcechunkStore): + return store.session + except ImportError: + pass + raise UnsupportedEngineError( + f"the zarrista async engine cannot serve a {type(store).__name__}; " + f"supported: {_ASYNC_SUPPORTED}." + ) diff --git a/tests/zarrista/__init__.py b/tests/zarrista/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/zarrista/test_engine.py b/tests/zarrista/test_engine.py new file mode 100644 index 0000000000..148899525a --- /dev/null +++ b/tests/zarrista/test_engine.py @@ -0,0 +1,101 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np +import pytest + +pytest.importorskip("zarrista") + +import zarr +from zarr.errors import UnsupportedEngineError +from zarr.storage import LocalStore + +if TYPE_CHECKING: + from pathlib import Path + + +def _make(tmp_path: Path) -> zarr.Array[Any]: + z = zarr.create_array(LocalStore(tmp_path), shape=(10, 9), chunks=(3, 4), dtype="float32") + z[:, :] = np.arange(90, dtype="float32").reshape(10, 9) + return z + + +def test_zarrista_engine_read_write_combinations(tmp_path: Path) -> None: + z = _make(tmp_path) + ze = zarr.open_array(LocalStore(tmp_path), engine="zarrista") + + # contiguous read + np.testing.assert_array_equal(np.asarray(ze[2:7, 1:5]), np.asarray(z[2:7, 1:5])) + # strided read via bbox + post-index + np.testing.assert_array_equal(np.asarray(ze[1:9:2, ::3]), np.asarray(z[1:9:2, ::3])) + # full-chunk-aligned write + ze[0:3, 0:4] = np.zeros((3, 4), dtype="float32") + np.testing.assert_array_equal(np.asarray(z[0:3, 0:4]), np.zeros((3, 4), dtype="float32")) + # partial-chunk RMW write + ze[1:2, 1:2] = np.float32(99.0) + assert float(np.asarray(z[1, 1])) == 99.0 + assert float(np.asarray(z[0, 0])) == 0.0 # neighbor in same chunk untouched + + +def test_zarrista_engine_edge_chunk_full_write(tmp_path: Path) -> None: + # shape (10, 9) with chunks (3, 4): row-chunk index 3 (rows 9:10) is an + # edge chunk whose *clipped* extent (1 row) is smaller than the nominal + # chunk shape (3 rows). Writing the entirety of that chunk's valid + # (clipped) region is not the same as writing the full nominal chunk, so + # the engine must still take the RMW path (rather than treating it as a + # "full chunk" write and handing zarrista a wrongly-shaped buffer). + z = _make(tmp_path) + ze = zarr.open_array(LocalStore(tmp_path), engine="zarrista") + + ze[9:10, 0:4] = -np.ones((1, 4), dtype="float32") + np.testing.assert_array_equal(np.asarray(z[9:10, 0:4]), -np.ones((1, 4), dtype="float32")) + # the preceding (non-edge) chunk must be untouched + np.testing.assert_array_equal( + np.asarray(z[6:9, 0:4]), np.arange(90, dtype="float32").reshape(10, 9)[6:9, 0:4] + ) + + +def test_zarrista_rejects_v2(tmp_path: Path) -> None: + zarr.create_array(LocalStore(tmp_path), shape=(4,), chunks=(2,), dtype="int8", zarr_format=2) + with pytest.raises(UnsupportedEngineError, match="v3"): + zarr.open_array(LocalStore(tmp_path), engine="zarrista") + + +def test_zarrista_vlen_read_not_implemented(tmp_path: Path) -> None: + # zarrista decodes vlen dtypes (e.g. strings) to a `VariableArray`, which + # only exposes the Arrow C Data interface -- `np.asarray` on it does not + # raise, it silently produces a wrong-shape 0-d `object` array. The engine + # must reject this itself rather than let a bogus read through. + z = zarr.create_array(LocalStore(tmp_path), shape=(4,), chunks=(2,), dtype="str") + z[:] = np.array(["a", "bb", "ccc", "dddd"], dtype=object) + ze = zarr.open_array(LocalStore(tmp_path), engine="zarrista") + with pytest.raises(NotImplementedError, match="VariableArray"): + ze[:] + + +async def test_zarrista_async_engine_read_write_combinations(tmp_path: Path) -> None: + # exercises `ZarristaAsyncEngine` directly (over an obstore-backed + # `ObjectStore`), rather than through the sync `Array`/`ZarristaEngine` + # path the other tests in this module cover. + obstore = pytest.importorskip("obstore") + from zarr.api import asynchronous as async_api + from zarr.storage import ObjectStore + + store = ObjectStore(obstore.store.LocalStore(prefix=str(tmp_path))) + z = await async_api.create_array(store=store, shape=(10, 9), chunks=(3, 4), dtype="float32") + await z.setitem((slice(None), slice(None)), np.arange(90, dtype="float32").reshape(10, 9)) + ze = await async_api.open_array(store=store, engine="zarrista") + + # contiguous read + out = await ze.getitem((slice(2, 7), slice(1, 5))) + expected = await z.getitem((slice(2, 7), slice(1, 5))) + np.testing.assert_array_equal(np.asarray(out), np.asarray(expected)) + # full-chunk-aligned write + await ze.setitem((slice(0, 3), slice(0, 4)), np.zeros((3, 4), dtype="float32")) + written = await z.getitem((slice(0, 3), slice(0, 4))) + np.testing.assert_array_equal(np.asarray(written), np.zeros((3, 4), dtype="float32")) + # partial-chunk RMW write + await ze.setitem((slice(1, 2), slice(1, 2)), np.float32(99.0)) + assert float(np.asarray(await z.getitem((1, 1)))) == 99.0 + assert float(np.asarray(await z.getitem((0, 0)))) == 0.0 # untouched neighbor diff --git a/tests/zarrista/test_translate.py b/tests/zarrista/test_translate.py new file mode 100644 index 0000000000..0c47a8c924 --- /dev/null +++ b/tests/zarrista/test_translate.py @@ -0,0 +1,41 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import pytest + +pytest.importorskip("zarrista") + +import zarrista + +from zarr.errors import UnsupportedEngineError +from zarr.storage import LocalStore, MemoryStore +from zarr.zarrista._translate import translate_store_async, translate_store_sync + +if TYPE_CHECKING: + from pathlib import Path + + +def test_local_store_translates_to_filesystem_store(tmp_path: Path) -> None: + zs = translate_store_sync(LocalStore(tmp_path)) + assert isinstance(zs, zarrista.store.FilesystemStore) + + +def test_memory_store_rejected_sync(tmp_path: Path) -> None: + with pytest.raises(UnsupportedEngineError, match="MemoryStore"): + translate_store_sync(MemoryStore()) + + +def test_local_store_rejected_async(tmp_path: Path) -> None: + # async side wants obstore/icechunk; LocalStore is sync-only in v1 + with pytest.raises(UnsupportedEngineError): + translate_store_async(LocalStore(tmp_path)) + + +def test_object_store_translates_to_obstore(tmp_path: Path) -> None: + obstore = pytest.importorskip("obstore") + from zarr.storage import ObjectStore + + inner = obstore.store.LocalStore(prefix=str(tmp_path)) + zstore = ObjectStore(inner) + assert translate_store_async(zstore) is inner diff --git a/uv.lock b/uv.lock index 838feef237..ccc1f008b9 100644 --- a/uv.lock +++ b/uv.lock @@ -1157,6 +1157,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0a/1e/8222edaee03c37350eaa726213614e343a62f1e56396dd000ad9277bfa3d/hypothesis-6.152.7-py3-none-any.whl", hash = "sha256:c0b17dd428fcb6e962f60315f6f4a77816c72fbb281ce9ba73699dabead5ec82", size = 533802, upload-time = "2026-05-13T04:19:30.635Z" }, ] +[[package]] +name = "icechunk" +version = "2.1.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "zarr" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/89/41/ea6d2ebe3fced2be5ca4e8540b776d9544a31e30f0a90fe7968b28288aff/icechunk-2.1.1.tar.gz", hash = "sha256:a46637e2e079edcfb92a6cc2a7a0cfb990224b0be2fb1931f967361d17498fa0", size = 3528133, upload-time = "2026-07-08T20:30:01.408Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b7/9c/9cad0e3c6f4ab671816e513a10ddde6c13ca5b290ee38ae4f6611b098263/icechunk-2.1.1-cp312-abi3-macosx_10_12_x86_64.whl", hash = "sha256:660e8ad231cddad0f86cd26cc2f46a845d6b5cfed7bf023e9ceefbdb78b7d3e4", size = 17092044, upload-time = "2026-07-08T20:30:09.175Z" }, + { url = "https://files.pythonhosted.org/packages/79/ba/e7af165c67a8707e18f5e97233093a9d13153948334788f168002bfabffd/icechunk-2.1.1-cp312-abi3-macosx_11_0_arm64.whl", hash = "sha256:8fe37a901d7ad6a6bedb074e60a521505cce11d418777b43d974601a53c03ed5", size = 15776756, upload-time = "2026-07-08T20:30:06.79Z" }, + { url = "https://files.pythonhosted.org/packages/63/5b/b02968cd97ef2d910dd46eadf03e078f630ddf14c869c3b0c05363e6739a/icechunk-2.1.1-cp312-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9131213e1d831e63a0f74d20792934f3e1c97d4084680788df0648f4c19cb985", size = 17473456, upload-time = "2026-07-08T20:30:04.128Z" }, + { url = "https://files.pythonhosted.org/packages/c8/59/d1a379c7206011da4d89ab32064ff98398eaf4cc32227189cfb443d252a1/icechunk-2.1.1-cp312-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:34cc925a3d339dde989889d6f703bed3eb0d5d2e41887cf12a485c36122159eb", size = 17117003, upload-time = "2026-07-08T20:29:59.159Z" }, + { url = "https://files.pythonhosted.org/packages/ca/08/dcab15706126a1ac05639d6213ce34fdf72ff5c77a85542e09a9fcab4356/icechunk-2.1.1-cp312-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:2ceef9b8b0d4988b35c33411e5c2da15364611ba8a88292f6d382fe6d216f452", size = 17338644, upload-time = "2026-07-08T20:30:11.397Z" }, + { url = "https://files.pythonhosted.org/packages/ed/93/6ca99877532b9f214ed62ead6c9921b5798d0ce1eed468ab0f71331bcd77/icechunk-2.1.1-cp312-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:f6bbcb216f636fc6b53682297a6438eeb81f577e976b430ba58407d05f34bb31", size = 17886257, upload-time = "2026-07-08T20:30:13.761Z" }, + { url = "https://files.pythonhosted.org/packages/5b/ca/14d4a288343b2e01e06888a86640e8ce3e16b6f2b8bd10616add382f71d4/icechunk-2.1.1-cp312-abi3-win_amd64.whl", hash = "sha256:b87329b7088301777f5dde7f6f4180938cf410768e4cb49d3b9cc064eda2b036", size = 16209937, upload-time = "2026-07-08T20:30:16.41Z" }, +] + [[package]] name = "idna" version = "3.15" @@ -2257,52 +2275,39 @@ wheels = [ [[package]] name = "obstore" -version = "0.9.4" +version = "0.11.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "typing-extensions", marker = "python_full_version < '3.13'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/d3/7a/3a37b0bf0da898478029fcc511a0d2a7252689b1f29e46db7ae74a219c74/obstore-0.9.4.tar.gz", hash = "sha256:e2b93f1372c59da2c7e74122fc6dc4b713d84fd4528b5b500ef7f548425496b5", size = 124167, upload-time = "2026-04-22T19:51:05.261Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/ac/25/4449a0066796b91e282d7604a66387bba399b14752598c748ea9557c4c32/obstore-0.9.4-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:0d17cd04e7f22960050a85f8daa6e274d693e8fb3b97b81eeaa293c6f9e62eb4", size = 4090743, upload-time = "2026-04-22T19:49:26.461Z" }, - { url = "https://files.pythonhosted.org/packages/93/91/639fe5f5644593b9f4bea66f8f29c7bfd4de3b3381fb74b4f7df678f505f/obstore-0.9.4-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:d4beec92710fb8826fb357baf28fb79a91ee07dcdfe73777207aa762164aaa35", size = 3876313, upload-time = "2026-04-22T19:49:28.107Z" }, - { url = "https://files.pythonhosted.org/packages/ce/71/d6675f845ebe1e3927f2dce6a2a4d5a393359274762ee00c5e6855d5f468/obstore-0.9.4-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d523c8c365ab60afb8d232614a00a92bea439a9f5c55b92486c23a47af038a1e", size = 4029950, upload-time = "2026-04-22T19:49:30.279Z" }, - { url = "https://files.pythonhosted.org/packages/0e/3a/5915a173f5c6a95f9ec186a7e29b0ce6a23bd9b04c2b0b29a351dbe2baf6/obstore-0.9.4-cp311-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:ee0483619088337ee365cb344fceee337e2670ec4de2a1da92ac7f6b2220f18e", size = 4129455, upload-time = "2026-04-22T19:49:31.934Z" }, - { url = "https://files.pythonhosted.org/packages/b5/a9/63c31d2d436c06c4d39ed5cb154fe54202b303854532ec09537c4ce0755b/obstore-0.9.4-cp311-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:83da348bf0a7dd84e5839c0cd54d79dcd08e0729c394e566f73a605b93b9e998", size = 4416727, upload-time = "2026-04-22T19:49:34.016Z" }, - { url = "https://files.pythonhosted.org/packages/7f/fa/23c5c6db02be0e13abcbe01c1ca94c5f7876e8c58e74cb9ac2b57b068866/obstore-0.9.4-cp311-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f282a17200bcc37b8d7a1d02a146ed41812eb6e76fd0a4c9a154f02da1b8031f", size = 4311520, upload-time = "2026-04-22T19:49:35.905Z" }, - { url = "https://files.pythonhosted.org/packages/86/f0/49f6b02dab9c05e3fd79d6129e4d9e7e9874d6e5e05369ca3b3b80a48aaa/obstore-0.9.4-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:d29dcfceaa0a205ded2263d29a2a3aa206819d549e0325c1f2106f79e2658584", size = 4220536, upload-time = "2026-04-22T19:49:38.343Z" }, - { url = "https://files.pythonhosted.org/packages/50/ab/d0bfd6d68422e7d8f2204d91736c7e62767e0576ad749da442a71e7773b2/obstore-0.9.4-cp311-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:caecb912723ab8e9da8da26def249d66da4318959df2bafc0a55af64f3255902", size = 4105099, upload-time = "2026-04-22T19:49:40.384Z" }, - { url = "https://files.pythonhosted.org/packages/66/3b/f595d0ee354f9daa69438991f8818602f34bc59498c8468456a02d45fb27/obstore-0.9.4-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0c1c06fec8837595a2829b5f7536d0d01e940ce10b07ad2a8594fec1cfd0b7d5", size = 4294206, upload-time = "2026-04-22T19:49:42.016Z" }, - { url = "https://files.pythonhosted.org/packages/60/54/3c5af2d59258aaa9e5bef05320658ea6e9b1f3897a3a977bf7f54a0b6ec1/obstore-0.9.4-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:c132795a789ec5ade31bf4d5b55ed321fb41d9749e9145520bf19063e1da5f7b", size = 4265047, upload-time = "2026-04-22T19:49:43.983Z" }, - { url = "https://files.pythonhosted.org/packages/fb/af/a8ba1feb81b9833b253147839da40405ec6bfa51feb3abfe909c800208a5/obstore-0.9.4-cp311-abi3-musllinux_1_2_i686.whl", hash = "sha256:c6e342360a5d0ae71486bc5f8311778aa144ec1a905c23593f8ef57b5bceae24", size = 4255361, upload-time = "2026-04-22T19:49:45.864Z" }, - { url = "https://files.pythonhosted.org/packages/15/f7/3ccc0288111e057f8ba3d99bee14f95d9e9bb00acaf6e9700e0eb4cd82c3/obstore-0.9.4-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:aeb6f7e7e862550f5020a10692ef6f02d5ba4912dba08942eb59bb7d73f93fe0", size = 4439378, upload-time = "2026-04-22T19:49:47.581Z" }, - { url = "https://files.pythonhosted.org/packages/c4/b1/3ac8b5772743c60064f3c7e02d27f346dbb58feaa99a49ee09798d1cfb00/obstore-0.9.4-cp311-abi3-win_amd64.whl", hash = "sha256:a58ef942292841f99d69ac11d19d05544c835447c8c09dacbfb7409c6374c4a1", size = 4191594, upload-time = "2026-04-22T19:49:49.308Z" }, - { url = "https://files.pythonhosted.org/packages/9d/81/8f6b6509f8df603261cdb5ddb521c49891457775669c6ad857812bf4a7c1/obstore-0.9.4-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:fff17f59390ed307afcd1fb18c56076c1f911dd9f5c2636b7d7133c4d07f8c3f", size = 4071300, upload-time = "2026-04-22T19:49:51.386Z" }, - { url = "https://files.pythonhosted.org/packages/ab/fe/0c74ddf3ab9b24ef356925bfb613bc7846f869220361a784b63f754d8563/obstore-0.9.4-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:4527c4c7889f1bd1f1952017d74774870e14e199d6b50b9e72f291f9498d898c", size = 3870593, upload-time = "2026-04-22T19:49:53.481Z" }, - { url = "https://files.pythonhosted.org/packages/73/fa/260ec94f9a7b4f4c8afbdd016710bed0736615488d3ac0c5620f9179bfcd/obstore-0.9.4-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a57c2016e3e569de35050f95c679ffe61813c4e3cb6d6028c4c3f57231021eb4", size = 4023990, upload-time = "2026-04-22T19:49:55.644Z" }, - { url = "https://files.pythonhosted.org/packages/8d/84/5b8e2b9607fb93c96a39a4cfa6d37bd3049ebf7265d0e9f8afa938bf32fe/obstore-0.9.4-cp313-cp313t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:bd5327cee4fb3578b51beb1c92915cc3a05ffe794be40f50bd68d27e97d78c5c", size = 4119971, upload-time = "2026-04-22T19:49:57.745Z" }, - { url = "https://files.pythonhosted.org/packages/3c/2b/e6c093acb7e62009d5b1678d82839903287c29d4a6e1dfbea8fbf41313d5/obstore-0.9.4-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:12b1e6105eafe02d8973dbeb2d274eeac2271c67f1126ffa16f18ddea8dd5443", size = 4407147, upload-time = "2026-04-22T19:49:59.928Z" }, - { url = "https://files.pythonhosted.org/packages/ce/3d/5c93a9adee8f045b89d5f21b337f53667499db770bda129f805723ab14e4/obstore-0.9.4-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:0b0d378248fda4e36652808d73eaaeb7e67154427e6c724248c9b0b9b03e70a6", size = 4312215, upload-time = "2026-04-22T19:50:01.534Z" }, - { url = "https://files.pythonhosted.org/packages/9a/de/507f60b4e6a8c0cad9f93a51a7b28132c9db49e20aadbcd542fa2abc57c4/obstore-0.9.4-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a78fb77c346abd2bcdfa071d7166be2bdc38c28573ae5a230746df6158a5593e", size = 4216936, upload-time = "2026-04-22T19:50:03.244Z" }, - { url = "https://files.pythonhosted.org/packages/7b/ff/612bd5f8258349bfe9e8c349d184b5ea3333038d4cce0d003eefafb2160c/obstore-0.9.4-cp313-cp313t-manylinux_2_24_aarch64.whl", hash = "sha256:f4e5a6dfe6877fb599868d560d6fcf4d7416cadbdf3bd947254b53830c2f11c0", size = 4105091, upload-time = "2026-04-22T19:50:05.038Z" }, - { url = "https://files.pythonhosted.org/packages/e3/73/b083b99e7bc0b529bee7b4437cafd7cc7d9f59c10995a48b6c26447fdf7f/obstore-0.9.4-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:f8114a2b84268c991232d89b105d9239299b6afb56e4941a61c09f3a89033022", size = 4292570, upload-time = "2026-04-22T19:50:06.823Z" }, - { url = "https://files.pythonhosted.org/packages/8b/cd/3c4555f98db9a49432bc0afa68bfc33dd47bdfa3699c915b4b0e887577e3/obstore-0.9.4-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:9d7b959f5f74532a142fb449c0bef5814dfe3fa5c43c31ac4284a15221a75aaf", size = 4261946, upload-time = "2026-04-22T19:50:08.789Z" }, - { url = "https://files.pythonhosted.org/packages/96/f8/bdc66df3d0dfdcfb3931a585a7fb3b74336619baf6d3540b1425b424232b/obstore-0.9.4-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:a8e9101fc2659dd938e7ae06512075bc0a8f02ab28d2ee438d6fca8b4f3bdfba", size = 4245595, upload-time = "2026-04-22T19:50:10.765Z" }, - { url = "https://files.pythonhosted.org/packages/d7/22/1aa58ea676293e5b888391c8433ff6ab8f66622aae30427287f9daac6d46/obstore-0.9.4-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:538384255545b5c575497fcab26389c8f01707402b6ddcdd73b769b66311635d", size = 4436599, upload-time = "2026-04-22T19:50:12.585Z" }, - { url = "https://files.pythonhosted.org/packages/1d/9e/b52f2c97be27952d488cf1980af0c635f9947003e5744e3e1dc6252f0040/obstore-0.9.4-cp313-cp313t-win_amd64.whl", hash = "sha256:eef1c772657bb1293adad0d671ca1ff1e1dcae84ec4dfbf1a34e47c2a1f134ac", size = 4180463, upload-time = "2026-04-22T19:50:14.288Z" }, - { url = "https://files.pythonhosted.org/packages/19/76/c53583f95c6811057abd3116756dca46785318d564a0e99c207cbb2d8938/obstore-0.9.4-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:e009e7437770c85beae4c32cb79f662f0a9922676ef127e943d107a5c082d38d", size = 4071302, upload-time = "2026-04-22T19:50:15.967Z" }, - { url = "https://files.pythonhosted.org/packages/2f/23/ac3b9c05a09b3d5f178ed6f288c5d6913df8f7386059590194e0fee65d15/obstore-0.9.4-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ac5f3ad314bd4592fe484b79c229518be7bb5f6218bed33c20742026d5caf860", size = 3870813, upload-time = "2026-04-22T19:50:17.62Z" }, - { url = "https://files.pythonhosted.org/packages/f7/ec/c3458e0f24d2d1a4f185f541905b07e51c91b3fec589b1600c77d511e585/obstore-0.9.4-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:db79d5ebc4177360565ffcec4abd49930cf052cdbeb94e3a3ece2e2d08f087d0", size = 4024237, upload-time = "2026-04-22T19:50:19.81Z" }, - { url = "https://files.pythonhosted.org/packages/a7/eb/6cf468a200e491fdc6c04075e2fbbac1707bbecd243f0f56ae1e75d052ed/obstore-0.9.4-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:05b565d89c3115fb74385852dd628e12f6645a1bba97523dceae016b538a3f33", size = 4119635, upload-time = "2026-04-22T19:50:21.605Z" }, - { url = "https://files.pythonhosted.org/packages/81/fb/b44d002767fa5af95ab4ca8e16c3a9057fc11f13de03f498b99adf0c4e50/obstore-0.9.4-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7dfc4fc98403d8fbb316eb04257c8122b6f1dda37e80869491fdacf60a815e4c", size = 4406906, upload-time = "2026-04-22T19:50:23.654Z" }, - { url = "https://files.pythonhosted.org/packages/4b/18/9a75ad5082cd581c4a55f0e62bedf4b030a8b53824976fc1f030eff225b3/obstore-0.9.4-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c69af620fd3d06a8cfb62d25faf1adb6ccc97cc572f47ee04dddcde5a5e5444e", size = 4311826, upload-time = "2026-04-22T19:50:25.458Z" }, - { url = "https://files.pythonhosted.org/packages/8d/03/b0f945b31f40364a7ed4dbc5677abc66331fcf478732f4d643e17e56bb13/obstore-0.9.4-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fa78c0230e0b9d49b25ed18980e1751331ddfe05782d6ce97579a9ccda8229ea", size = 4217086, upload-time = "2026-04-22T19:50:27.266Z" }, - { url = "https://files.pythonhosted.org/packages/1a/26/bdd85264c806802086f21d73cc7c95a5baca5feeeac4bce8acb97142163f/obstore-0.9.4-cp314-cp314t-manylinux_2_24_aarch64.whl", hash = "sha256:c828719f0bb310a9cf0e0f08cb62a0b8cc550138617cb03ac897900aec9d3d47", size = 4105560, upload-time = "2026-04-22T19:50:29.324Z" }, - { url = "https://files.pythonhosted.org/packages/6a/36/4a4a6a398e5f145edd1886388ebe5e6f6bbaf74950a5dea1a6ceae63e6b5/obstore-0.9.4-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:49a0455519f284b6bc2e0694298114926aff1d1f3d5d344e9163e03b446826cc", size = 4292582, upload-time = "2026-04-22T19:50:31.028Z" }, - { url = "https://files.pythonhosted.org/packages/39/4c/9caa197cd2eba726e9a5285db34027049b9527a23e1a7e08479678ad6a4a/obstore-0.9.4-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:cf437309fc0fe852591ae50405300490229f876ea06574651fd753ca3fd23f25", size = 4261613, upload-time = "2026-04-22T19:50:32.868Z" }, - { url = "https://files.pythonhosted.org/packages/b8/94/a3fbe6fb3ee1c57fd4943ddbb21848eea3925b77e0789614c857d86b795e/obstore-0.9.4-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:d83dbd20b6a5d42e35794ef64046de39040854829ec4f1eb2f6dfb54df48cc3d", size = 4245638, upload-time = "2026-04-22T19:50:35.009Z" }, - { url = "https://files.pythonhosted.org/packages/56/a7/d18e168f318327d63512dfa7cf3b5e89ed9bfba6d6a8917ad7d4700b8657/obstore-0.9.4-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:5a0c337f37f30a2d66555d69bf3abd840457a279c57ede93bd02e014721ed364", size = 4437226, upload-time = "2026-04-22T19:50:36.635Z" }, - { url = "https://files.pythonhosted.org/packages/f4/ce/66aadd155db1e273c6ec2236c0fb904666d10c2e3b791b40624c272e586c/obstore-0.9.4-cp314-cp314t-win_amd64.whl", hash = "sha256:24e37a1c713c95a964e119f8ef879415a495432162e74e80ed29d645aeeca114", size = 4180746, upload-time = "2026-04-22T19:50:38.396Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/2e/2f/f83afaab7945509d72245b2b00af0b4834ce78fdd2d9ae9f0ad1a3036a91/obstore-0.11.0.tar.gz", hash = "sha256:a2f55163bcd348b4a60d12e6893eac50eddc742bad8032a1705d49140b992204", size = 130565, upload-time = "2026-06-25T18:29:49.405Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fc/b2/00c213e7e5ca8065f97e37e55294adab836e3f6a88b23e4029069aaecf95/obstore-0.11.0-cp311-abi3-macosx_10_12_x86_64.whl", hash = "sha256:42f36546c7ac44dbab1173d2330a8a1b1a3f0e37950e553b8c904e3dd0744b25", size = 5491935, upload-time = "2026-06-25T18:28:32.029Z" }, + { url = "https://files.pythonhosted.org/packages/ac/37/6a6b9a5e15a8a37c24d14317a87648097c4888593b588510c03c030d2e90/obstore-0.11.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:687bb9d3962d568b7c439c5d0c6fea19b2749862a8e5c8eebd0c058c4eccde9e", size = 4672619, upload-time = "2026-06-25T18:28:33.852Z" }, + { url = "https://files.pythonhosted.org/packages/28/f9/6745ce8c4f7bfac19dc14a4438b48a2e93a689b92b0cecfc695e41a4e8b1/obstore-0.11.0-cp311-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:010b51578c7514a41719d795cdb7a1e6529be509dac3772e477187a59422bb97", size = 5072806, upload-time = "2026-06-25T18:28:36.127Z" }, + { url = "https://files.pythonhosted.org/packages/6c/18/991d3b3cdd851c0225e55f3dc45b47fd9e249827d188995011469f805132/obstore-0.11.0-cp311-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:cfaa8129a3f5d8518a3a75184d4b02348db0f6263177cd1f0951f6568243cc9e", size = 5303777, upload-time = "2026-06-25T18:28:37.89Z" }, + { url = "https://files.pythonhosted.org/packages/8d/e9/90e56015a45b5e56a84fc3188c4e5fb088b288d41992c73a629e10df6760/obstore-0.11.0-cp311-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:c790a5cb9ff2970d1f464a6a708d734dce9939e9f668cb6708c5dba5d61589b2", size = 5493871, upload-time = "2026-06-25T18:28:39.981Z" }, + { url = "https://files.pythonhosted.org/packages/66/02/f1744091d59ce71c5523174eb860fbb298275c901e89b9ea6fbf3e654a33/obstore-0.11.0-cp311-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:827113e12fe8088e0281a9d57b90b2b8dbc8a6ffe3b15dadb9baa5feb3d266c1", size = 5361913, upload-time = "2026-06-25T18:28:42.089Z" }, + { url = "https://files.pythonhosted.org/packages/5d/59/3f47822683ee2b6db8685faa25829946d6343a561251ec2704548455d946/obstore-0.11.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a2ff6d3ed553298828fb760b4aef6347fbcc7b5c5e3ce3f8381ce805c370021a", size = 5638724, upload-time = "2026-06-25T18:28:43.897Z" }, + { url = "https://files.pythonhosted.org/packages/23/50/1df335fdf9b527b3933f1e94ab6fc720ad314260fab8591cb0b6668ff192/obstore-0.11.0-cp311-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:39d04b324fcf984e7050734ebda77b81764025b0c011750201a0d8954087f7aa", size = 5413508, upload-time = "2026-06-25T18:28:45.624Z" }, + { url = "https://files.pythonhosted.org/packages/de/dc/a259aba149b841ca7c91fea177df9972a60a636b54077beed1a35b254994/obstore-0.11.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:37c0d15d775b1370ef5204ee3919a5ddf7e2592d11815213105f8db031f2ab8d", size = 5619995, upload-time = "2026-06-25T18:28:47.599Z" }, + { url = "https://files.pythonhosted.org/packages/e5/b4/ec25fdb4d6b060bc6eea647fc0e88f75fcc20fe8d16d67fb0dbe999d323b/obstore-0.11.0-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:7f468caf9b6e0f12ff151e5fe618de5fc9192befa9bd02734b06de4efd2e49f6", size = 5299512, upload-time = "2026-06-25T18:28:49.629Z" }, + { url = "https://files.pythonhosted.org/packages/a8/e5/29be060d06ec13e2af3d1b6cfb77b7c37f8be6c56b77295c945fefad73e4/obstore-0.11.0-cp311-abi3-musllinux_1_2_i686.whl", hash = "sha256:42d8e8fad85be8ee488c1a9a9b7c6a42128abb84e67175da40d3d1165c1846df", size = 5427026, upload-time = "2026-06-25T18:28:51.317Z" }, + { url = "https://files.pythonhosted.org/packages/57/b7/577a965f440e9ea64243518663f9d16be7df8eafc7123818e8e841fa21ce/obstore-0.11.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:9c8fd2a544e2e0b926669c47fcfb8d2314e234abc240ea165dae04ee42e1d7ac", size = 5869187, upload-time = "2026-06-25T18:28:53.166Z" }, + { url = "https://files.pythonhosted.org/packages/e2/18/8fdbaee22bfd5b9c44e1fdff8ca0508e2fe60c42bf9fc85f0c9c27b4ecf2/obstore-0.11.0-cp311-abi3-win_amd64.whl", hash = "sha256:6fb3d4678c0f4242d3109362e9b1df5d7b27765f43d5aacb2e81af53a75cb9ef", size = 5329384, upload-time = "2026-06-25T18:28:55.305Z" }, + { url = "https://files.pythonhosted.org/packages/8b/8b/7555e48ec768728fcfc71a051c6b28d6ddaf1bececf492ce5ef995aab5f0/obstore-0.11.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:f3132393eff9f3f2b543ecbb3bcc12319a7c433fef06493b4350d6854d505a14", size = 5515763, upload-time = "2026-06-25T18:28:57.527Z" }, + { url = "https://files.pythonhosted.org/packages/23/8f/94d83f3336421cbb5e436ab0ae5695eae72f7c82990d6b1ac090712c8052/obstore-0.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a3a8da19b47af4c14ecc694209b3c18ac6d89f96be5656ee3a19b77947c14155", size = 4649491, upload-time = "2026-06-25T18:28:59.386Z" }, + { url = "https://files.pythonhosted.org/packages/fe/86/11f4e1f51a8c6cf21a5915c018d2357201ad3c5799d418f0c6529fafaab2/obstore-0.11.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e91298b9b6c3a0408c28eece62bca6c5b6cda2f6350351d84e07b4dc8fb2631f", size = 5060659, upload-time = "2026-06-25T18:29:01.139Z" }, + { url = "https://files.pythonhosted.org/packages/49/e7/fd3036b0923d10e878e2073020f1ef692a618ed1cc3980d3e4a468c93713/obstore-0.11.0-cp314-cp314t-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3422c532486671dfb5e3e739bf15ec9ca2a8da544a3c23b74ae3857dcab1c6a8", size = 5277058, upload-time = "2026-06-25T18:29:02.96Z" }, + { url = "https://files.pythonhosted.org/packages/2f/a7/b016c3ac6857ac856326dc0a292e3871b8500d03f25f3b90e168b05de357/obstore-0.11.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3de027ce46cf0592c2b41654e2c27dbc52a4a726f6fbc511380acc0da3a9f658", size = 5475852, upload-time = "2026-06-25T18:29:05.032Z" }, + { url = "https://files.pythonhosted.org/packages/85/9e/644ffe8db7757de7f71f94a036ab24222bccb4de290d3ca69f76547e812d/obstore-0.11.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:1a80a95548678210bc336b866e37139c293565c3b163eb3fef2433d5d6640a33", size = 5363082, upload-time = "2026-06-25T18:29:06.878Z" }, + { url = "https://files.pythonhosted.org/packages/ae/0b/26af6b5fa6ba96af84086f44f78b4e5b0af1729c31402d7b28c68989d174/obstore-0.11.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:acaa261dc15efb95bbeca06f8fe9b47ee23d7302a6aa1fa3a9654baab8b23d7c", size = 5629116, upload-time = "2026-06-25T18:29:08.771Z" }, + { url = "https://files.pythonhosted.org/packages/ae/ce/f30d502991c6719b2fbd7b8385ef3e39da07bfca099108bcc5eeed8b9c20/obstore-0.11.0-cp314-cp314t-manylinux_2_24_aarch64.whl", hash = "sha256:a3300cabbc3129670987b3723629c791d83c117ef1b6a0c670c2043648e000a1", size = 5404534, upload-time = "2026-06-25T18:29:11.294Z" }, + { url = "https://files.pythonhosted.org/packages/52/20/d5bf5f816e868717ba647ed9a2109e800deb402d0265d410456b3fcb4376/obstore-0.11.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:f4e6a9480843645cd4ee122c41d5c5a46a56f1e9cdda85638826f2e0e439fe5c", size = 5613159, upload-time = "2026-06-25T18:29:13.414Z" }, + { url = "https://files.pythonhosted.org/packages/db/b4/6d4c1c211e3b06cc8554189e0d4406e8fa1f98ed55f9213b8e398a11599f/obstore-0.11.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:9fb2b1814c4314b8903f4e2ebbe8c3365fea6543669615ee9a0288b0d3a2edeb", size = 5286279, upload-time = "2026-06-25T18:29:15.424Z" }, + { url = "https://files.pythonhosted.org/packages/02/5e/d7b5589424a56171b16ab94cf92eb493490c300aaa044913bbdd94cace68/obstore-0.11.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:63fb9b072815eafe4705f617f567d896b1c858adcb390795fa1e269367791031", size = 5401780, upload-time = "2026-06-25T18:29:17.514Z" }, + { url = "https://files.pythonhosted.org/packages/c4/18/841baea8936e51a18b0e5d4c51f09c0a7798cb73b027e9794be2362a0f0b/obstore-0.11.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:086fafba314ff98cfab1c4bf7814699e862513e8889720cf6f7462296cb32787", size = 5853618, upload-time = "2026-06-25T18:29:19.354Z" }, + { url = "https://files.pythonhosted.org/packages/83/9a/d6127f5422b78e0222b0a9eadcfd7a5aa8d873a9498da7d4a77d4ac8ce2e/obstore-0.11.0-cp314-cp314t-win_amd64.whl", hash = "sha256:676d1154f6f08721110f9b7d14ee3a3c0293abaf9da135bb90f54e276dca1cac", size = 5314113, upload-time = "2026-06-25T18:29:21.209Z" }, ] [[package]] @@ -4057,6 +4062,23 @@ test = [ { name = "tomlkit" }, { name = "uv" }, ] +zarrista = [ + { name = "coverage" }, + { name = "hypothesis" }, + { name = "icechunk" }, + { name = "numpydoc" }, + { name = "obstore" }, + { name = "pytest" }, + { name = "pytest-accept" }, + { name = "pytest-asyncio" }, + { name = "pytest-benchmark" }, + { name = "pytest-codspeed" }, + { name = "pytest-cov" }, + { name = "pytest-xdist" }, + { name = "tomlkit" }, + { name = "uv" }, + { name = "zarrista" }, +] zarrs = [ { name = "coverage" }, { name = "hypothesis" }, @@ -4177,6 +4199,23 @@ test = [ { name = "tomlkit" }, { name = "uv" }, ] +zarrista = [ + { name = "coverage", specifier = ">=7.10" }, + { name = "hypothesis" }, + { name = "icechunk", specifier = ">=1.1.21" }, + { name = "numpydoc" }, + { name = "obstore", specifier = ">=0.10.1" }, + { name = "pytest" }, + { name = "pytest-accept" }, + { name = "pytest-asyncio" }, + { name = "pytest-benchmark" }, + { name = "pytest-codspeed" }, + { name = "pytest-cov" }, + { name = "pytest-xdist" }, + { name = "tomlkit" }, + { name = "uv" }, + { name = "zarrista", git = "https://github.com/developmentseed/zarrista?rev=95e47ad4c414c5920f0cf15550f923039641da8e" }, +] zarrs = [ { name = "coverage", specifier = ">=7.10" }, { name = "hypothesis" }, @@ -4193,6 +4232,26 @@ zarrs = [ { name = "zarrs-bindings", directory = "packages/zarrs-bindings" }, ] +[[package]] +name = "zarr-metadata" +version = "0.3.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/f2/9c/cfd5aa02a27c63ecec702a77834b395411518da5c748414d7e6a323638ed/zarr_metadata-0.3.0.tar.gz", hash = "sha256:d8fe02feef43380056ea0429ceb50974b7b5afe6f0386853977506b034e89d53", size = 36398, upload-time = "2026-06-19T13:17:38.276Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/91/96/64137930fb40b96b4d207eb1f1e4e42c5d6c9e682a5a7c3e6feff6eb0e29/zarr_metadata-0.3.0-py3-none-any.whl", hash = "sha256:e6651f418fcc89cc3c6fc11aa852fb6f8dd6f31d62913abc0d4d37ce7302d671", size = 45636, upload-time = "2026-06-19T13:17:37.097Z" }, +] + +[[package]] +name = "zarrista" +version = "0.1.0b7" +source = { git = "https://github.com/developmentseed/zarrista?rev=95e47ad4c414c5920f0cf15550f923039641da8e#95e47ad4c414c5920f0cf15550f923039641da8e" } +dependencies = [ + { name = "zarr-metadata" }, +] + [[package]] name = "zarrs-bindings" source = { directory = "packages/zarrs-bindings" } From 8249ab02656a2111758b1af6100e28084d6c63c2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 20:58:29 +0200 Subject: [PATCH 59/75] refactor!: remove zarr.crud, zarr.zarrs, and the zarrs-bindings crate Delete the superseded modules (zarr.crud, zarr.zarrs) and the bundled Rust zarrs-bindings crate which have been fully replaced by the engine layer (Tasks 1-8). Removes crud config registration and pytest ignores, updates pyproject.toml to remove zarrs dependency group and uv source configuration, and deletes the zarrs workflow. Assisted-by: ClaudeCode:claude-haiku-4-5 --- .github/workflows/zarrs.yml | 34 - changes/4064.feature.md | 9 - packages/zarrs-bindings/Cargo.lock | 1831 ------------------------ packages/zarrs-bindings/Cargo.toml | 21 - packages/zarrs-bindings/pyproject.toml | 14 - packages/zarrs-bindings/src/chunk.rs | 168 --- packages/zarrs-bindings/src/lib.rs | 52 - packages/zarrs-bindings/src/node.rs | 122 -- packages/zarrs-bindings/src/store.rs | 225 --- pyproject.toml | 9 - src/zarr/core/config.py | 1 - src/zarr/crud/__init__.py | 55 - src/zarr/crud/_api.py | 344 ----- src/zarr/crud/_backend.py | 76 - src/zarr/crud/_common.py | 21 - src/zarr/crud/_reference.py | 203 --- src/zarr/crud/_registry.py | 40 - src/zarr/zarrs/__init__.py | 26 - src/zarr/zarrs/_backend.py | 150 -- src/zarr/zarrs/_bridge.py | 105 -- tests/crud/__init__.py | 0 tests/crud/conftest.py | 95 -- tests/crud/test_crud.py | 319 ----- tests/crud/test_reference_backend.py | 73 - tests/crud/test_registry.py | 39 - tests/zarrs/__init__.py | 0 tests/zarrs/conftest.py | 46 - tests/zarrs/test_bridge.py | 53 - tests/zarrs/test_cache.py | 87 -- uv.lock | 34 - 30 files changed, 4252 deletions(-) delete mode 100644 .github/workflows/zarrs.yml delete mode 100644 changes/4064.feature.md delete mode 100644 packages/zarrs-bindings/Cargo.lock delete mode 100644 packages/zarrs-bindings/Cargo.toml delete mode 100644 packages/zarrs-bindings/pyproject.toml delete mode 100644 packages/zarrs-bindings/src/chunk.rs delete mode 100644 packages/zarrs-bindings/src/lib.rs delete mode 100644 packages/zarrs-bindings/src/node.rs delete mode 100644 packages/zarrs-bindings/src/store.rs delete mode 100644 src/zarr/crud/__init__.py delete mode 100644 src/zarr/crud/_api.py delete mode 100644 src/zarr/crud/_backend.py delete mode 100644 src/zarr/crud/_common.py delete mode 100644 src/zarr/crud/_reference.py delete mode 100644 src/zarr/crud/_registry.py delete mode 100644 src/zarr/zarrs/__init__.py delete mode 100644 src/zarr/zarrs/_backend.py delete mode 100644 src/zarr/zarrs/_bridge.py delete mode 100644 tests/crud/__init__.py delete mode 100644 tests/crud/conftest.py delete mode 100644 tests/crud/test_crud.py delete mode 100644 tests/crud/test_reference_backend.py delete mode 100644 tests/crud/test_registry.py delete mode 100644 tests/zarrs/__init__.py delete mode 100644 tests/zarrs/conftest.py delete mode 100644 tests/zarrs/test_bridge.py delete mode 100644 tests/zarrs/test_cache.py diff --git a/.github/workflows/zarrs.yml b/.github/workflows/zarrs.yml deleted file mode 100644 index b17cfb3a3b..0000000000 --- a/.github/workflows/zarrs.yml +++ /dev/null @@ -1,34 +0,0 @@ -name: Zarrs bindings - -on: - push: - branches: [ main ] - pull_request: - branches: [ main ] - workflow_dispatch: - -permissions: - contents: read - -concurrency: - group: ${{ github.workflow }}-${{ github.ref }} - cancel-in-progress: true - -jobs: - test: - runs-on: ubuntu-latest - steps: - - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - with: - fetch-depth: 0 # hatch-vcs needs tags to compute zarr's version - persist-credentials: false - - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - with: - python-version: '3.12' - - name: Install Rust - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable - - name: Run zarrs bindings tests - # the ubuntu runner image ships a Rust toolchain; the maturin build - # backend is fetched by uv on demand - run: uv run --group zarrs pytest tests/crud tests/zarrs -v diff --git a/changes/4064.feature.md b/changes/4064.feature.md deleted file mode 100644 index 26216a71ae..0000000000 --- a/changes/4064.feature.md +++ /dev/null @@ -1,9 +0,0 @@ -Added `zarr.crud`, an experimental backend-agnostic low-level functional API for -zarr hierarchy CRUD (`create_*`, `read_chunk`, `read_region`, `read_encoded_chunk`, -`write_chunk`, `delete_chunk`, `read_metadata`, `delete_node`, `list_children`). -Array routines take an explicit metadata document, enabling read-only views. -Operations delegate to a pluggable `CrudBackend`: a pure-Python reference backend -(the default) or the zarrs-accelerated backend in `zarr.zarrs`, backed by the Rust -[zarrs](https://zarrs.dev) crate via the in-repo `zarrs-bindings` PyO3 crate. -Select a backend with the `crud.backend` config key or a per-call `backend=` -argument. Build the zarrs backend for development with `uv sync --group zarrs`. diff --git a/packages/zarrs-bindings/Cargo.lock b/packages/zarrs-bindings/Cargo.lock deleted file mode 100644 index 9c9a91f203..0000000000 --- a/packages/zarrs-bindings/Cargo.lock +++ /dev/null @@ -1,1831 +0,0 @@ -# This file is automatically @generated by Cargo. -# It is not intended for manual editing. -version = 4 - -[[package]] -name = "adler2" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" - -[[package]] -name = "ahash" -version = "0.8.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" -dependencies = [ - "cfg-if", - "getrandom 0.3.4", - "once_cell", - "version_check", - "zerocopy", -] - -[[package]] -name = "aho-corasick" -version = "1.1.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" -dependencies = [ - "memchr", -] - -[[package]] -name = "allocator-api2" -version = "0.2.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" - -[[package]] -name = "anyhow" -version = "1.0.102" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" - -[[package]] -name = "async-lock" -version = "3.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "290f7f2596bd5b78a9fec8088ccd89180d7f9f55b94b0576823bbbdc72ee8311" -dependencies = [ - "event-listener", - "event-listener-strategy", - "pin-project-lite", -] - -[[package]] -name = "async-trait" -version = "0.1.89" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "auto_impl" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ffdcb70bdbc4d478427380519163274ac86e52916e10f0a8889adf0f96d3fee7" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "autocfg" -version = "1.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" - -[[package]] -name = "base64" -version = "0.22.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" - -[[package]] -name = "bitflags" -version = "2.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" - -[[package]] -name = "blosc-src" -version = "0.3.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a9046dd58971db0226346fde214143d16a6eb12f535b5320d0ea94fcea420631" -dependencies = [ - "cc", - "libz-sys", - "lz4-sys", - "snappy_src", - "zstd-sys", -] - -[[package]] -name = "blusc" -version = "0.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b4e0c17eaa785d2673fe58c22fc817946c2330ed47f3d9f79835d65950d32a45" -dependencies = [ - "flate2", - "lz4_flex", - "pkg-config", - "snap", - "zstd", -] - -[[package]] -name = "bumpalo" -version = "3.20.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" - -[[package]] -name = "bytemuck" -version = "1.25.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec" -dependencies = [ - "bytemuck_derive", -] - -[[package]] -name = "bytemuck_derive" -version = "1.10.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "byteorder" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" - -[[package]] -name = "bytes" -version = "1.11.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" - -[[package]] -name = "cc" -version = "1.2.63" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f" -dependencies = [ - "find-msvc-tools", - "jobserver", - "libc", - "shlex", -] - -[[package]] -name = "cfg-if" -version = "1.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" - -[[package]] -name = "concurrent-queue" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4ca0197aee26d1ae37445ee532fefce43251d24cc7c166799f4d46817f1d3973" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "convert_case" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "633458d4ef8c78b72454de2d54fd6ab2e60f9e02be22f3c6104cdc8a4e0fceb9" -dependencies = [ - "unicode-segmentation", -] - -[[package]] -name = "crc32c" -version = "0.6.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a47af21622d091a8f0fb295b88bc886ac74efcc613efc19f5d0b21de5c89e47" -dependencies = [ - "rustc_version", -] - -[[package]] -name = "crc32fast" -version = "1.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "crossbeam-channel" -version = "0.5.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82b8f8f868b36967f9606790d1903570de9ceaf870a7bf9fbbd3016d636a2cb2" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-deque" -version = "0.8.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9dd111b7b7f7d55b72c0a6ae361660ee5853c9af73f70c3c2ef6858b950e2e51" -dependencies = [ - "crossbeam-epoch", - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-epoch" -version = "0.9.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b82ac4a3c2ca9c3460964f020e1402edd5753411d7737aa39c3714ad1b5420e" -dependencies = [ - "crossbeam-utils", -] - -[[package]] -name = "crossbeam-utils" -version = "0.8.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" - -[[package]] -name = "crunchy" -version = "0.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" - -[[package]] -name = "derive_more" -version = "2.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d751e9e49156b02b44f9c1815bcb94b984cdcc4396ecc32521c739452808b134" -dependencies = [ - "derive_more-impl", -] - -[[package]] -name = "derive_more-impl" -version = "2.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "799a97264921d8623a957f6c3b9011f3b5492f557bbb7a5a19b7fa6d06ba8dcb" -dependencies = [ - "convert_case", - "proc-macro2", - "quote", - "rustc_version", - "syn", - "unicode-xid", -] - -[[package]] -name = "either" -version = "1.16.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" - -[[package]] -name = "equivalent" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" - -[[package]] -name = "event-listener" -version = "5.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e13b66accf52311f30a0db42147dadea9850cb48cd070028831ae5f5d4b856ab" -dependencies = [ - "concurrent-queue", - "parking", - "pin-project-lite", -] - -[[package]] -name = "event-listener-strategy" -version = "0.5.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8be9f3dfaaffdae2972880079a491a1a8bb7cbed0b8dd7a347f668b4150a3b93" -dependencies = [ - "event-listener", - "pin-project-lite", -] - -[[package]] -name = "find-msvc-tools" -version = "0.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" - -[[package]] -name = "flate2" -version = "1.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" -dependencies = [ - "crc32fast", - "miniz_oxide", -] - -[[package]] -name = "foldhash" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" - -[[package]] -name = "foldhash" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" - -[[package]] -name = "futures" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" -dependencies = [ - "futures-channel", - "futures-core", - "futures-executor", - "futures-io", - "futures-sink", - "futures-task", - "futures-util", -] - -[[package]] -name = "futures-channel" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" -dependencies = [ - "futures-core", - "futures-sink", -] - -[[package]] -name = "futures-core" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" - -[[package]] -name = "futures-executor" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" -dependencies = [ - "futures-core", - "futures-task", - "futures-util", -] - -[[package]] -name = "futures-io" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" - -[[package]] -name = "futures-macro" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "futures-sink" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" - -[[package]] -name = "futures-task" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" - -[[package]] -name = "futures-util" -version = "0.3.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" -dependencies = [ - "futures-channel", - "futures-core", - "futures-io", - "futures-macro", - "futures-sink", - "futures-task", - "memchr", - "pin-project-lite", - "slab", -] - -[[package]] -name = "getrandom" -version = "0.3.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" -dependencies = [ - "cfg-if", - "js-sys", - "libc", - "r-efi 5.3.0", - "wasip2", - "wasm-bindgen", -] - -[[package]] -name = "getrandom" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" -dependencies = [ - "cfg-if", - "libc", - "r-efi 6.0.0", - "wasip2", - "wasip3", -] - -[[package]] -name = "half" -version = "2.7.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" -dependencies = [ - "bytemuck", - "cfg-if", - "crunchy", - "zerocopy", -] - -[[package]] -name = "hashbrown" -version = "0.15.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" -dependencies = [ - "allocator-api2", - "equivalent", - "foldhash 0.1.5", -] - -[[package]] -name = "hashbrown" -version = "0.16.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" -dependencies = [ - "allocator-api2", - "equivalent", - "foldhash 0.2.0", -] - -[[package]] -name = "hashbrown" -version = "0.17.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" - -[[package]] -name = "heck" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" - -[[package]] -name = "id-arena" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" - -[[package]] -name = "indexmap" -version = "2.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" -dependencies = [ - "equivalent", - "hashbrown 0.17.1", - "serde", - "serde_core", -] - -[[package]] -name = "inventory" -version = "0.3.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a4f0c30c76f2f4ccee3fe55a2435f691ca00c0e4bd87abe4f4a851b1d4dac39b" -dependencies = [ - "rustversion", -] - -[[package]] -name = "itertools" -version = "0.14.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" -dependencies = [ - "either", -] - -[[package]] -name = "itoa" -version = "1.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" - -[[package]] -name = "jobserver" -version = "0.1.34" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" -dependencies = [ - "getrandom 0.3.4", - "libc", -] - -[[package]] -name = "js-sys" -version = "0.3.100" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f2025f20d7a4fa7785846e7b63d10a76d3f1cee98ee5cb79ea59703f95e42162" -dependencies = [ - "cfg-if", - "futures-util", - "wasm-bindgen", -] - -[[package]] -name = "leb128fmt" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" - -[[package]] -name = "libc" -version = "0.2.186" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" - -[[package]] -name = "libz-sys" -version = "1.1.29" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85bc9657773828b90eeb625adff10eeac83cc21bbfd8e23a03eaa8a33c9e28d9" -dependencies = [ - "cc", - "libc", - "pkg-config", - "vcpkg", -] - -[[package]] -name = "link-cplusplus" -version = "1.0.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f78c730aaa7d0b9336a299029ea49f9ee53b0ed06e9202e8cb7db9bae7b8c82" -dependencies = [ - "cc", -] - -[[package]] -name = "lock_api" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" -dependencies = [ - "scopeguard", -] - -[[package]] -name = "log" -version = "0.4.32" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" - -[[package]] -name = "lru" -version = "0.12.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "234cf4f4a04dc1f57e24b96cc0cd600cf2af460d4161ac5ecdd0af8e1f3b2a38" -dependencies = [ - "hashbrown 0.15.5", -] - -[[package]] -name = "lru" -version = "0.16.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f66e8d5d03f609abc3a39e6f08e4164ebf1447a732906d39eb9b99b7919ef39" -dependencies = [ - "hashbrown 0.16.1", -] - -[[package]] -name = "lz4-sys" -version = "1.11.1+lz4-1.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6bd8c0d6c6ed0cd30b3652886bb8711dc4bb01d637a68105a3d5158039b418e6" -dependencies = [ - "cc", - "libc", -] - -[[package]] -name = "lz4_flex" -version = "0.12.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "90071f8077f8e40adfc4b7fe9cd495ce316263f19e75c2211eeff3fdf475a3d9" -dependencies = [ - "twox-hash", -] - -[[package]] -name = "matrixmultiply" -version = "0.3.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a06de3016e9fae57a36fd14dba131fccf49f74b40b7fbdb472f96e361ec71a08" -dependencies = [ - "autocfg", - "rawpointer", -] - -[[package]] -name = "memchr" -version = "2.8.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" - -[[package]] -name = "miniz_oxide" -version = "0.8.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" -dependencies = [ - "adler2", - "simd-adler32", -] - -[[package]] -name = "moka" -version = "0.12.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046" -dependencies = [ - "crossbeam-channel", - "crossbeam-epoch", - "crossbeam-utils", - "equivalent", - "parking_lot", - "portable-atomic", - "smallvec", - "tagptr", - "uuid", -] - -[[package]] -name = "monostate" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb4cc965c89dd0615a9e822ff8002f7633d2466143d51bd58693e4b2c75aabad" -dependencies = [ - "monostate-impl", - "serde", - "serde_core", -] - -[[package]] -name = "monostate-impl" -version = "1.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "23f5b99488110875b5904839d396c2cdfaf241ff6622638acb879cc7effad5de" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "ndarray" -version = "0.17.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "520080814a7a6b4a6e9070823bb24b4531daac8c4627e08ba5de8c5ef2f2752d" -dependencies = [ - "matrixmultiply", - "num-complex", - "num-integer", - "num-traits", - "portable-atomic", - "portable-atomic-util", - "rawpointer", -] - -[[package]] -name = "num" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "35bd024e8b2ff75562e5f34e7f4905839deb4b22955ef5e73d2fea1b9813cb23" -dependencies = [ - "num-bigint", - "num-complex", - "num-integer", - "num-iter", - "num-rational", - "num-traits", -] - -[[package]] -name = "num-bigint" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" -dependencies = [ - "num-integer", - "num-traits", -] - -[[package]] -name = "num-complex" -version = "0.4.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" -dependencies = [ - "bytemuck", - "num-traits", -] - -[[package]] -name = "num-integer" -version = "0.1.46" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" -dependencies = [ - "num-traits", -] - -[[package]] -name = "num-iter" -version = "0.1.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf" -dependencies = [ - "autocfg", - "num-integer", - "num-traits", -] - -[[package]] -name = "num-rational" -version = "0.4.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f83d14da390562dca69fc84082e73e548e1ad308d24accdedd2720017cb37824" -dependencies = [ - "num-bigint", - "num-integer", - "num-traits", -] - -[[package]] -name = "num-traits" -version = "0.2.19" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" -dependencies = [ - "autocfg", -] - -[[package]] -name = "once_cell" -version = "1.21.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" - -[[package]] -name = "page_size" -version = "0.6.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30d5b2194ed13191c1999ae0704b7839fb18384fa22e49b57eeaa97d79ce40da" -dependencies = [ - "libc", - "winapi", -] - -[[package]] -name = "parking" -version = "2.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f38d5652c16fde515bb1ecef450ab0f6a219d619a7274976324d5e377f7dceba" - -[[package]] -name = "parking_lot" -version = "0.12.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" -dependencies = [ - "lock_api", - "parking_lot_core", -] - -[[package]] -name = "parking_lot_core" -version = "0.9.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" -dependencies = [ - "cfg-if", - "libc", - "redox_syscall", - "smallvec", - "windows-link", -] - -[[package]] -name = "paste" -version = "1.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" - -[[package]] -name = "pathdiff" -version = "0.2.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df94ce210e5bc13cb6651479fa48d14f601d9858cfe0467f43ae157023b938d3" - -[[package]] -name = "pin-project-lite" -version = "0.2.17" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" - -[[package]] -name = "pkg-config" -version = "0.3.33" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" - -[[package]] -name = "portable-atomic" -version = "1.13.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" - -[[package]] -name = "portable-atomic-util" -version = "0.2.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a106d1259c23fac8e543272398ae0e3c0b8d33c88ed73d0cc71b0f1d902618" -dependencies = [ - "portable-atomic", -] - -[[package]] -name = "positioned-io" -version = "0.3.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4ec4b80060f033312b99b6874025d9503d2af87aef2dd4c516e253fbfcdada7" -dependencies = [ - "byteorder", - "libc", - "winapi", -] - -[[package]] -name = "prettyplease" -version = "0.2.37" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" -dependencies = [ - "proc-macro2", - "syn", -] - -[[package]] -name = "proc-macro2" -version = "1.0.106" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "pyo3" -version = "0.28.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91fd8e38a3b50ed1167fb981cd6fd60147e091784c427b8f7183a7ee32c31c12" -dependencies = [ - "libc", - "once_cell", - "portable-atomic", - "pyo3-build-config", - "pyo3-ffi", - "pyo3-macros", -] - -[[package]] -name = "pyo3-build-config" -version = "0.28.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e368e7ddfdeb98c9bca7f8383be1648fd84ab466bf2bc015e94008db6d35611e" -dependencies = [ - "target-lexicon", -] - -[[package]] -name = "pyo3-ffi" -version = "0.28.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f29e10af80b1f7ccaf7f69eace800a03ecd13e883acfacc1e5d0988605f651e" -dependencies = [ - "libc", - "pyo3-build-config", -] - -[[package]] -name = "pyo3-macros" -version = "0.28.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "df6e520eff47c45997d2fc7dd8214b25dd1310918bbb2642156ef66a67f29813" -dependencies = [ - "proc-macro2", - "pyo3-macros-backend", - "quote", - "syn", -] - -[[package]] -name = "pyo3-macros-backend" -version = "0.28.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c4cdc218d835738f81c2338f822078af45b4afdf8b2e33cbb5916f108b813acb" -dependencies = [ - "heck", - "proc-macro2", - "pyo3-build-config", - "quote", - "syn", -] - -[[package]] -name = "quick_cache" -version = "0.6.23" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a3db184a8b66cfe87f0263a1de147a6b554c864d1767c6f7fa4eb0e5497b565" -dependencies = [ - "ahash", - "equivalent", - "hashbrown 0.16.1", - "parking_lot", -] - -[[package]] -name = "quote" -version = "1.0.45" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" -dependencies = [ - "proc-macro2", -] - -[[package]] -name = "r-efi" -version = "5.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" - -[[package]] -name = "r-efi" -version = "6.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" - -[[package]] -name = "rawpointer" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60a357793950651c4ed0f3f52338f53b2f809f32d83a07f72909fa13e4c6c1e3" - -[[package]] -name = "rayon" -version = "1.12.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "fb39b166781f92d482534ef4b4b1b2568f42613b53e5b6c160e24cfbfa30926d" -dependencies = [ - "either", - "rayon-core", -] - -[[package]] -name = "rayon-core" -version = "1.13.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "22e18b0f0062d30d4230b2e85ff77fdfe4326feb054b9783a3460d8435c8ab91" -dependencies = [ - "crossbeam-deque", - "crossbeam-utils", -] - -[[package]] -name = "rayon_iter_concurrent_limit" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d09ee01023de07fa073ce14c37cbe0a9e099c6b0b60a29cf4af6d04d9553fed7" -dependencies = [ - "rayon", -] - -[[package]] -name = "redox_syscall" -version = "0.5.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" -dependencies = [ - "bitflags", -] - -[[package]] -name = "regex" -version = "1.12.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba" -dependencies = [ - "aho-corasick", - "memchr", - "regex-automata", - "regex-syntax", -] - -[[package]] -name = "regex-automata" -version = "0.4.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" -dependencies = [ - "aho-corasick", - "memchr", - "regex-syntax", -] - -[[package]] -name = "regex-syntax" -version = "0.8.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" - -[[package]] -name = "rustc_version" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" -dependencies = [ - "semver", -] - -[[package]] -name = "rustversion" -version = "1.0.22" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" - -[[package]] -name = "same-file" -version = "1.0.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" -dependencies = [ - "winapi-util", -] - -[[package]] -name = "scopeguard" -version = "1.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" - -[[package]] -name = "semver" -version = "1.0.28" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" - -[[package]] -name = "serde" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" -dependencies = [ - "serde_core", - "serde_derive", -] - -[[package]] -name = "serde_core" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" -dependencies = [ - "serde_derive", -] - -[[package]] -name = "serde_derive" -version = "1.0.228" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "serde_json" -version = "1.0.150" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" -dependencies = [ - "indexmap", - "itoa", - "memchr", - "serde", - "serde_core", - "zmij", -] - -[[package]] -name = "serde_repr" -version = "0.1.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "shlex" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" - -[[package]] -name = "simd-adler32" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" - -[[package]] -name = "slab" -version = "0.4.12" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" - -[[package]] -name = "smallvec" -version = "1.15.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" - -[[package]] -name = "snap" -version = "1.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" - -[[package]] -name = "snappy_src" -version = "0.2.5+snappy.1.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4e1432067a55bcfb1fd522d2aca6537a4fcea32bba87ea86921226d14f9bad53" -dependencies = [ - "cc", - "link-cplusplus", -] - -[[package]] -name = "syn" -version = "2.0.117" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" -dependencies = [ - "proc-macro2", - "quote", - "unicode-ident", -] - -[[package]] -name = "tagptr" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" - -[[package]] -name = "target-lexicon" -version = "0.13.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" - -[[package]] -name = "thiserror" -version = "2.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" -dependencies = [ - "thiserror-impl", -] - -[[package]] -name = "thiserror-impl" -version = "2.0.18" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "thread_local" -version = "1.1.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185" -dependencies = [ - "cfg-if", -] - -[[package]] -name = "tinyvec" -version = "1.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" -dependencies = [ - "tinyvec_macros", -] - -[[package]] -name = "tinyvec_macros" -version = "0.1.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" - -[[package]] -name = "twox-hash" -version = "2.1.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" - -[[package]] -name = "unicode-ident" -version = "1.0.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" - -[[package]] -name = "unicode-segmentation" -version = "1.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" - -[[package]] -name = "unicode-xid" -version = "0.2.6" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" - -[[package]] -name = "unsafe_cell_slice" -version = "0.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6659959f702dcdaad77bd6e42a9409a32ceccc06943ec93c8a4306be00eb6cf1" - -[[package]] -name = "uuid" -version = "1.23.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" -dependencies = [ - "getrandom 0.4.2", - "js-sys", - "wasm-bindgen", -] - -[[package]] -name = "vcpkg" -version = "0.2.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "accd4ea62f7bb7a82fe23066fb0957d48ef677f6eeb8215f372f52e48bb32426" - -[[package]] -name = "version_check" -version = "0.9.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" - -[[package]] -name = "walkdir" -version = "2.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" -dependencies = [ - "same-file", - "winapi-util", -] - -[[package]] -name = "wasip2" -version = "1.0.3+wasi-0.2.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" -dependencies = [ - "wit-bindgen 0.57.1", -] - -[[package]] -name = "wasip3" -version = "0.4.0+wasi-0.3.0-rc-2026-01-06" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" -dependencies = [ - "wit-bindgen 0.51.0", -] - -[[package]] -name = "wasm-bindgen" -version = "0.2.123" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a254a4b10c19a76f09a27640e7ffbf9bc30bf67e16a3bf28aaefa4920fe81563" -dependencies = [ - "cfg-if", - "once_cell", - "rustversion", - "wasm-bindgen-macro", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-macro" -version = "0.2.123" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "24a40fc75b0ec6f3746ceb10d36f53a93dcd68a93b11b6445983945d79eba0dc" -dependencies = [ - "quote", - "wasm-bindgen-macro-support", -] - -[[package]] -name = "wasm-bindgen-macro-support" -version = "0.2.123" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "908f34bd9b9ce3d4caf07b72dfab63d61504d156856c6bd3cd87fa350cf3985b" -dependencies = [ - "bumpalo", - "proc-macro2", - "quote", - "syn", - "wasm-bindgen-shared", -] - -[[package]] -name = "wasm-bindgen-shared" -version = "0.2.123" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7acbf7616c27b194bbb550bf77ed0c2c3e5b7fd1260a93082b95fb7f47959b92" -dependencies = [ - "unicode-ident", -] - -[[package]] -name = "wasm-encoder" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" -dependencies = [ - "leb128fmt", - "wasmparser", -] - -[[package]] -name = "wasm-metadata" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" -dependencies = [ - "anyhow", - "indexmap", - "wasm-encoder", - "wasmparser", -] - -[[package]] -name = "wasmparser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" -dependencies = [ - "bitflags", - "hashbrown 0.15.5", - "indexmap", - "semver", -] - -[[package]] -name = "winapi" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5c839a674fcd7a98952e593242ea400abe93992746761e38641405d28b00f419" -dependencies = [ - "winapi-i686-pc-windows-gnu", - "winapi-x86_64-pc-windows-gnu", -] - -[[package]] -name = "winapi-i686-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ac3b87c63620426dd9b991e5ce0329eff545bccbbb34f3be09ff6fb6ab51b7b6" - -[[package]] -name = "winapi-util" -version = "0.1.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" -dependencies = [ - "windows-sys", -] - -[[package]] -name = "winapi-x86_64-pc-windows-gnu" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" - -[[package]] -name = "windows-link" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" - -[[package]] -name = "windows-sys" -version = "0.61.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" -dependencies = [ - "windows-link", -] - -[[package]] -name = "wit-bindgen" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" -dependencies = [ - "wit-bindgen-rust-macro", -] - -[[package]] -name = "wit-bindgen" -version = "0.57.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" - -[[package]] -name = "wit-bindgen-core" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" -dependencies = [ - "anyhow", - "heck", - "wit-parser", -] - -[[package]] -name = "wit-bindgen-rust" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" -dependencies = [ - "anyhow", - "heck", - "indexmap", - "prettyplease", - "syn", - "wasm-metadata", - "wit-bindgen-core", - "wit-component", -] - -[[package]] -name = "wit-bindgen-rust-macro" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" -dependencies = [ - "anyhow", - "prettyplease", - "proc-macro2", - "quote", - "syn", - "wit-bindgen-core", - "wit-bindgen-rust", -] - -[[package]] -name = "wit-component" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" -dependencies = [ - "anyhow", - "bitflags", - "indexmap", - "log", - "serde", - "serde_derive", - "serde_json", - "wasm-encoder", - "wasm-metadata", - "wasmparser", - "wit-parser", -] - -[[package]] -name = "wit-parser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" -dependencies = [ - "anyhow", - "id-arena", - "indexmap", - "log", - "semver", - "serde", - "serde_derive", - "serde_json", - "unicode-xid", - "wasmparser", -] - -[[package]] -name = "zarrs" -version = "0.23.13" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8132307b8fc041fd21f68c7987103fb6e038b11f9838c16ec43b798f5480ccf5" -dependencies = [ - "async-lock", - "base64", - "blosc-src", - "blusc", - "bytemuck", - "bytes", - "crc32c", - "derive_more", - "flate2", - "getrandom 0.3.4", - "half", - "inventory", - "itertools", - "itoa", - "libz-sys", - "log", - "lru 0.16.4", - "moka", - "ndarray", - "num", - "num-complex", - "paste", - "quick_cache", - "rayon", - "rayon_iter_concurrent_limit", - "serde", - "serde_json", - "thiserror", - "thread_local", - "unsafe_cell_slice", - "uuid", - "zarrs_chunk_grid", - "zarrs_chunk_key_encoding", - "zarrs_codec", - "zarrs_data_type", - "zarrs_filesystem", - "zarrs_metadata", - "zarrs_metadata_ext", - "zarrs_plugin", - "zarrs_storage", - "zstd", -] - -[[package]] -name = "zarrs-bindings" -version = "0.1.0" -dependencies = [ - "lru 0.12.5", - "pyo3", - "serde_json", - "zarrs", -] - -[[package]] -name = "zarrs_chunk_grid" -version = "0.5.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1cf67386fd96a0336cd3e5ab5ca6cb14e0e05aee80f1acae8c4d3cf562a8bb65" -dependencies = [ - "derive_more", - "inventory", - "itertools", - "rayon", - "thiserror", - "tinyvec", - "zarrs_metadata", - "zarrs_plugin", -] - -[[package]] -name = "zarrs_chunk_key_encoding" -version = "0.2.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9040e7feaa92d1904d492acd0cd91b97214f1791c5b5738e6c05b2ca4145a382" -dependencies = [ - "derive_more", - "inventory", - "zarrs_metadata", - "zarrs_plugin", - "zarrs_storage", -] - -[[package]] -name = "zarrs_codec" -version = "0.2.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "383a129a6a0cbb2c80cdba23809e5cab85159756464b7d0f112468a495c128da" -dependencies = [ - "async-trait", - "bytemuck", - "derive_more", - "futures", - "inventory", - "itertools", - "rayon", - "thiserror", - "unsafe_cell_slice", - "zarrs_chunk_grid", - "zarrs_data_type", - "zarrs_metadata", - "zarrs_plugin", - "zarrs_storage", -] - -[[package]] -name = "zarrs_data_type" -version = "0.9.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc7c594c9363278fcd9db4c205514f009944206eb093ea7ad40b85f50009f31" -dependencies = [ - "derive_more", - "half", - "inventory", - "num", - "paste", - "serde", - "serde_json", - "thiserror", - "zarrs_metadata", - "zarrs_plugin", -] - -[[package]] -name = "zarrs_filesystem" -version = "0.3.9" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "270efeb0181651aee5460b3232f2fc83e91bd646cefe75001d1c8f9a4f3abf81" -dependencies = [ - "bytes", - "derive_more", - "itertools", - "libc", - "page_size", - "pathdiff", - "positioned-io", - "thiserror", - "walkdir", - "zarrs_storage", -] - -[[package]] -name = "zarrs_metadata" -version = "0.7.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d60c4c363a8a302d7babb3c29017850a7b4e0af6ca5f9ba2946263a185b62fea" -dependencies = [ - "derive_more", - "half", - "monostate", - "serde", - "serde_json", - "thiserror", -] - -[[package]] -name = "zarrs_metadata_ext" -version = "0.4.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2048e07848ca99c7450518e0584929300b1b6a3cf442f18b26ffd3520814bd5b" -dependencies = [ - "derive_more", - "monostate", - "num", - "serde", - "serde_json", - "serde_repr", - "thiserror", - "zarrs_metadata", -] - -[[package]] -name = "zarrs_plugin" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5cbe0ed432aee86856f70ca33be36eaf4a0dae21ab730750d9280a7ca1e95046" -dependencies = [ - "paste", - "regex", - "serde_json", - "thiserror", -] - -[[package]] -name = "zarrs_storage" -version = "0.4.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d098796d2ed4cf94896569615101e0432e870a7665396da5cc32300fb68f7c1" -dependencies = [ - "auto_impl", - "bytes", - "derive_more", - "itertools", - "thiserror", - "unsafe_cell_slice", -] - -[[package]] -name = "zerocopy" -version = "0.8.52" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f" -dependencies = [ - "zerocopy-derive", -] - -[[package]] -name = "zerocopy-derive" -version = "0.8.52" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" -dependencies = [ - "proc-macro2", - "quote", - "syn", -] - -[[package]] -name = "zmij" -version = "1.0.21" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" - -[[package]] -name = "zstd" -version = "0.13.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" -dependencies = [ - "zstd-safe", -] - -[[package]] -name = "zstd-safe" -version = "7.2.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" -dependencies = [ - "zstd-sys", -] - -[[package]] -name = "zstd-sys" -version = "2.0.16+zstd.1.5.7" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" -dependencies = [ - "cc", - "pkg-config", -] diff --git a/packages/zarrs-bindings/Cargo.toml b/packages/zarrs-bindings/Cargo.toml deleted file mode 100644 index ab06ef3517..0000000000 --- a/packages/zarrs-bindings/Cargo.toml +++ /dev/null @@ -1,21 +0,0 @@ -[package] -name = "zarrs-bindings" -version = "0.1.0" -edition = "2024" -rust-version = "1.91" -publish = false -license = "MIT" -description = "PyO3 bindings to the zarrs Rust crate, consumed by zarr.zarrs" - -[lib] -name = "_zarrs_bindings" -crate-type = ["cdylib"] - -[dependencies] -lru = "0.12" -pyo3 = { version = "0.28", features = ["abi3-py312"] } -serde_json = "1" -zarrs = "0.23" - -[profile.release] -lto = "thin" diff --git a/packages/zarrs-bindings/pyproject.toml b/packages/zarrs-bindings/pyproject.toml deleted file mode 100644 index 4212a64b56..0000000000 --- a/packages/zarrs-bindings/pyproject.toml +++ /dev/null @@ -1,14 +0,0 @@ -[build-system] -requires = ["maturin>=1.7,<2"] -build-backend = "maturin" - -[project] -name = "zarrs-bindings" -dynamic = ["version"] -description = "PyO3 bindings to the zarrs Rust crate, consumed by zarr.zarrs" -requires-python = ">=3.12" -license = "MIT" - -[tool.maturin] -module-name = "_zarrs_bindings" -strip = true diff --git a/packages/zarrs-bindings/src/chunk.rs b/packages/zarrs-bindings/src/chunk.rs deleted file mode 100644 index 556995924b..0000000000 --- a/packages/zarrs-bindings/src/chunk.rs +++ /dev/null @@ -1,168 +0,0 @@ -use std::num::NonZeroUsize; -use std::sync::{Arc, Mutex, OnceLock}; - -use lru::LruCache; -use pyo3::exceptions::PyNotImplementedError; -use pyo3::prelude::*; -use pyo3::types::PyBytes; -use zarrs::array::{Array, ArrayBytes, ArraySubset}; -use zarrs::metadata::ArrayMetadata; -use zarrs::storage::ReadableWritableListableStorage; - -use crate::store::resolve_store_with_key; -use crate::{runtime_err, value_err}; - -type DynArray = Array; - -/// Cache of constructed Arrays keyed by (filesystem root, node path, metadata -/// JSON). Only native filesystem stores are cached (see `resolve_store_with_key`). -/// Bounded by an LRU; entries hold only a filesystem path + codec chain, no data. -type CacheKey = (String, String, String); -static ARRAY_CACHE: OnceLock>>> = OnceLock::new(); - -fn array_cache() -> &'static Mutex>> { - ARRAY_CACHE.get_or_init(|| Mutex::new(LruCache::new(NonZeroUsize::new(128).unwrap()))) -} - -/// Acquire the array cache lock, recovering gracefully from a poisoned mutex -/// (e.g. a thread panicked while holding it). The worst case is a stale or -/// partially-updated cache entry — far preferable to wedging all array I/O. -fn lock_cache() -> std::sync::MutexGuard<'static, LruCache>> { - array_cache().lock().unwrap_or_else(|e| e.into_inner()) -} - -fn build_array( - storage: ReadableWritableListableStorage, - path: &str, - metadata_json: &str, -) -> PyResult { - let metadata = ArrayMetadata::try_from(metadata_json).map_err(value_err)?; - Array::new_with_metadata(storage, path, metadata).map_err(value_err) -} - -/// Construct (or fetch from cache) an Array view from an explicit metadata -/// document, without consulting the store for metadata. When `cache_key` is -/// `Some(root)` the result is memoized on (root, path, metadata_json). -fn array_view( - storage: ReadableWritableListableStorage, - cache_key: Option, - path: &str, - metadata_json: &str, -) -> PyResult> { - if let Some(root) = cache_key { - let key = (root, path.to_string(), metadata_json.to_string()); - if let Some(array) = lock_cache().get(&key).cloned() { - return Ok(array); - } - let array = Arc::new(build_array(storage, path, metadata_json)?); - lock_cache().put(key, Arc::clone(&array)); - Ok(array) - } else { - Ok(Arc::new(build_array(storage, path, metadata_json)?)) - } -} - -#[pyfunction] -pub(crate) fn array_cache_len() -> usize { - lock_cache().len() -} - -#[pyfunction] -pub(crate) fn clear_array_cache() { - lock_cache().clear(); -} - -#[pyfunction] -pub(crate) fn retrieve_chunk( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - chunk_coords: Vec, -) -> PyResult> { - let (storage, cache_key) = resolve_store_with_key(store)?; - let data = py.detach(move || -> PyResult> { - let array = array_view(storage, cache_key, &path, &metadata_json)?; - let bytes: ArrayBytes<'static> = - array.retrieve_chunk(&chunk_coords).map_err(runtime_err)?; - let fixed = bytes.into_fixed().map_err(|_| { - PyNotImplementedError::new_err("variable-length data types are not supported") - })?; - Ok(fixed.into_owned()) - })?; - Ok(PyBytes::new(py, &data).unbind()) -} - -#[pyfunction] -pub(crate) fn retrieve_encoded_chunk( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - chunk_coords: Vec, -) -> PyResult>> { - let (storage, cache_key) = resolve_store_with_key(store)?; - let data = py.detach(move || -> PyResult>> { - let array = array_view(storage, cache_key, &path, &metadata_json)?; - array - .retrieve_encoded_chunk(&chunk_coords) - .map_err(runtime_err) - })?; - Ok(data.map(|d| PyBytes::new(py, &d).unbind())) -} - -#[pyfunction] -pub(crate) fn store_chunk( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - chunk_coords: Vec, - data: Vec, -) -> PyResult<()> { - let (storage, cache_key) = resolve_store_with_key(store)?; - py.detach(move || { - let array = array_view(storage, cache_key, &path, &metadata_json)?; - array - .store_chunk(&chunk_coords, ArrayBytes::new_flen(data)) - .map_err(runtime_err) - }) -} - -#[pyfunction] -pub(crate) fn erase_chunk( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - chunk_coords: Vec, -) -> PyResult<()> { - let (storage, cache_key) = resolve_store_with_key(store)?; - py.detach(move || { - let array = array_view(storage, cache_key, &path, &metadata_json)?; - array.erase_chunk(&chunk_coords).map_err(runtime_err) - }) -} - -#[pyfunction] -pub(crate) fn retrieve_array_subset( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - start: Vec, - shape: Vec, -) -> PyResult> { - let (storage, cache_key) = resolve_store_with_key(store)?; - let data = py.detach(move || -> PyResult> { - let array = array_view(storage, cache_key, &path, &metadata_json)?; - let subset = ArraySubset::new_with_start_shape(start, shape).map_err(value_err)?; - let bytes: ArrayBytes<'static> = - array.retrieve_array_subset(&subset).map_err(runtime_err)?; - let fixed = bytes.into_fixed().map_err(|_| { - PyNotImplementedError::new_err("variable-length data types are not supported") - })?; - Ok(fixed.into_owned()) - })?; - Ok(PyBytes::new(py, &data).unbind()) -} diff --git a/packages/zarrs-bindings/src/lib.rs b/packages/zarrs-bindings/src/lib.rs deleted file mode 100644 index 61f947480f..0000000000 --- a/packages/zarrs-bindings/src/lib.rs +++ /dev/null @@ -1,52 +0,0 @@ -use pyo3::exceptions::{PyRuntimeError, PyValueError}; -use pyo3::prelude::*; - -mod chunk; -mod node; -mod store; - -pyo3::create_exception!( - _zarrs_bindings, - NodeExistsError, - PyValueError, - "A node already exists at the given path." -); -pyo3::create_exception!( - _zarrs_bindings, - NodeNotFoundError, - PyValueError, - "No node was found at the given path." -); - -pub(crate) fn runtime_err(err: impl std::fmt::Display) -> PyErr { - PyRuntimeError::new_err(err.to_string()) -} - -pub(crate) fn value_err(err: impl std::fmt::Display) -> PyErr { - PyValueError::new_err(err.to_string()) -} - -#[pyfunction] -fn version() -> &'static str { - env!("CARGO_PKG_VERSION") -} - -#[pymodule] -fn _zarrs_bindings(m: &Bound<'_, PyModule>) -> PyResult<()> { - m.add("NodeExistsError", m.py().get_type::())?; - m.add("NodeNotFoundError", m.py().get_type::())?; - m.add_function(wrap_pyfunction!(version, m)?)?; - m.add_function(wrap_pyfunction!(node::create_array, m)?)?; - m.add_function(wrap_pyfunction!(node::create_group, m)?)?; - m.add_function(wrap_pyfunction!(node::delete_node, m)?)?; - m.add_function(wrap_pyfunction!(node::list_children, m)?)?; - m.add_function(wrap_pyfunction!(node::read_metadata, m)?)?; - m.add_function(wrap_pyfunction!(chunk::retrieve_chunk, m)?)?; - m.add_function(wrap_pyfunction!(chunk::retrieve_encoded_chunk, m)?)?; - m.add_function(wrap_pyfunction!(chunk::store_chunk, m)?)?; - m.add_function(wrap_pyfunction!(chunk::erase_chunk, m)?)?; - m.add_function(wrap_pyfunction!(chunk::retrieve_array_subset, m)?)?; - m.add_function(wrap_pyfunction!(chunk::array_cache_len, m)?)?; - m.add_function(wrap_pyfunction!(chunk::clear_array_cache, m)?)?; - Ok(()) -} diff --git a/packages/zarrs-bindings/src/node.rs b/packages/zarrs-bindings/src/node.rs deleted file mode 100644 index 35a057ab31..0000000000 --- a/packages/zarrs-bindings/src/node.rs +++ /dev/null @@ -1,122 +0,0 @@ -use pyo3::prelude::*; -use zarrs::array::Array; -use zarrs::group::Group; -use zarrs::metadata::{ArrayMetadata, GroupMetadata}; -use zarrs::node::{Node, NodePath, node_exists}; -use zarrs::storage::{ReadableWritableListableStorage, StorePrefix}; - -use crate::store::resolve_store; -use crate::{NodeExistsError, NodeNotFoundError, runtime_err, value_err}; - -/// `path` arguments throughout this module are zarrs node paths, e.g. "/" or -/// "/foo/bar" (already normalized by the Python layer's `_node_path`). -pub(crate) fn parse_node_path(path: &str) -> PyResult { - NodePath::new(path).map_err(value_err) -} - -/// When a node exists at `node_path`: erase it (and everything under it) if -/// `overwrite`, otherwise raise `NodeExistsError`. -pub(crate) fn prepare_target( - storage: &ReadableWritableListableStorage, - node_path: &NodePath, - overwrite: bool, -) -> PyResult<()> { - if node_exists(storage, node_path).map_err(runtime_err)? { - if !overwrite { - return Err(NodeExistsError::new_err(format!( - "a node already exists at path {}", - node_path.as_str() - ))); - } - let prefix: StorePrefix = node_path.try_into().map_err(value_err)?; - storage.erase_prefix(&prefix).map_err(runtime_err)?; - } - Ok(()) -} - -#[pyfunction] -pub(crate) fn create_group( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - overwrite: bool, -) -> PyResult<()> { - let storage = resolve_store(store)?; - let metadata = GroupMetadata::try_from(metadata_json.as_str()).map_err(value_err)?; - py.detach(move || { - let node_path = parse_node_path(&path)?; - prepare_target(&storage, &node_path, overwrite)?; - let group = Group::new_with_metadata(storage, &path, metadata).map_err(value_err)?; - group.store_metadata().map_err(runtime_err) - }) -} - -#[pyfunction] -pub(crate) fn create_array( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, - metadata_json: String, - overwrite: bool, -) -> PyResult<()> { - let storage = resolve_store(store)?; - let metadata = ArrayMetadata::try_from(metadata_json.as_str()).map_err(value_err)?; - py.detach(move || { - let node_path = parse_node_path(&path)?; - prepare_target(&storage, &node_path, overwrite)?; - let array = Array::new_with_metadata(storage, &path, metadata).map_err(value_err)?; - array.store_metadata().map_err(runtime_err) - }) -} - -#[pyfunction] -pub(crate) fn read_metadata( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, -) -> PyResult { - let storage = resolve_store(store)?; - py.detach(move || { - let node = - Node::open(&storage, &path).map_err(|e| NodeNotFoundError::new_err(e.to_string()))?; - serde_json::to_string(node.metadata()).map_err(runtime_err) - }) -} - -#[pyfunction] -pub(crate) fn delete_node(py: Python<'_>, store: &Bound<'_, PyAny>, path: String) -> PyResult<()> { - let storage = resolve_store(store)?; - py.detach(move || { - let node_path = parse_node_path(&path)?; - if !node_exists(&storage, &node_path).map_err(runtime_err)? { - return Err(NodeNotFoundError::new_err(format!( - "no node found at path {}", - node_path.as_str() - ))); - } - let prefix: StorePrefix = (&node_path).try_into().map_err(value_err)?; - storage.erase_prefix(&prefix).map_err(runtime_err) - }) -} - -#[pyfunction] -pub(crate) fn list_children( - py: Python<'_>, - store: &Bound<'_, PyAny>, - path: String, -) -> PyResult> { - let storage = resolve_store(store)?; - py.detach(move || { - let group = - Group::open(storage, &path).map_err(|e| NodeNotFoundError::new_err(e.to_string()))?; - let children = group.children(false).map_err(runtime_err)?; - children - .into_iter() - .map(|node| { - let metadata = serde_json::to_string(node.metadata()).map_err(runtime_err)?; - Ok((node.path().as_str().to_string(), metadata)) - }) - .collect() - }) -} diff --git a/packages/zarrs-bindings/src/store.rs b/packages/zarrs-bindings/src/store.rs deleted file mode 100644 index c58de37387..0000000000 --- a/packages/zarrs-bindings/src/store.rs +++ /dev/null @@ -1,225 +0,0 @@ -use std::sync::Arc; - -use pyo3::exceptions::PyValueError; -use pyo3::prelude::*; -use pyo3::types::{PyBytes, PyDict}; -use zarrs::filesystem::FilesystemStore; -use zarrs::storage::byte_range::{ByteRange, ByteRangeIterator}; -use zarrs::storage::{ - Bytes, ListableStorageTraits, MaybeBytes, MaybeBytesIterator, OffsetBytesIterator, - ReadableStorageTraits, ReadableWritableListableStorage, StorageError, StoreKey, StoreKeys, - StoreKeysPrefixes, StorePrefix, WritableStorageTraits, -}; - -/// A zarrs store backed by a Python `zarr.zarrs._bridge.StoreShim`. -/// -/// Every method attaches to the Python interpreter and calls the shim, which -/// blocks on the zarr event loop. Blocking waits in Python release the GIL, so -/// the loop thread can make progress while a Rust worker waits here. -pub(crate) struct PyStore(Py); - -fn py_err(err: PyErr) -> StorageError { - StorageError::Other(err.to_string()) -} - -fn invalid(err: impl std::fmt::Display) -> StorageError { - StorageError::Other(err.to_string()) -} - -impl PyStore { - fn get_with_range( - &self, - key: &StoreKey, - range: Option<&ByteRange>, - ) -> Result { - Python::attach(|py| { - let shim = self.0.bind(py); - let result = match range { - None => shim.call_method1("get", (key.as_str(),)), - Some(ByteRange::FromStart(offset, length)) => { - shim.call_method1("get_range", (key.as_str(), *offset, *length)) - } - Some(ByteRange::Suffix(suffix)) => { - shim.call_method1("get_suffix", (key.as_str(), *suffix)) - } - } - .map_err(py_err)?; - if result.is_none() { - Ok(None) - } else { - let bytes: Vec = result.extract().map_err(py_err)?; - Ok(Some(Bytes::from(bytes))) - } - }) - } -} - -impl ReadableStorageTraits for PyStore { - fn get(&self, key: &StoreKey) -> Result { - self.get_with_range(key, None) - } - - fn get_partial_many<'a>( - &'a self, - key: &StoreKey, - byte_ranges: ByteRangeIterator<'a>, - ) -> Result, StorageError> { - let mut out = Vec::new(); - for byte_range in byte_ranges { - match self.get_with_range(key, Some(&byte_range))? { - Some(bytes) => out.push(Ok(bytes)), - None => return Ok(None), - } - } - Ok(Some(Box::new(out.into_iter()))) - } - - fn size_key(&self, key: &StoreKey) -> Result, StorageError> { - Python::attach(|py| { - self.0 - .bind(py) - .call_method1("getsize", (key.as_str(),)) - .map_err(py_err)? - .extract() - .map_err(py_err) - }) - } - - fn supports_get_partial(&self) -> bool { - true - } -} - -impl WritableStorageTraits for PyStore { - fn set(&self, key: &StoreKey, value: Bytes) -> Result<(), StorageError> { - Python::attach(|py| { - let data = PyBytes::new(py, &value); - self.0 - .bind(py) - .call_method1("set", (key.as_str(), data)) - .map_err(py_err)?; - Ok(()) - }) - } - - fn set_partial_many( - &self, - key: &StoreKey, - offset_values: OffsetBytesIterator, - ) -> Result<(), StorageError> { - // read-modify-write fallback provided by zarrs - zarrs::storage::store_set_partial_many(self, key, offset_values) - } - - fn supports_set_partial(&self) -> bool { - false - } - - fn erase(&self, key: &StoreKey) -> Result<(), StorageError> { - Python::attach(|py| { - self.0 - .bind(py) - .call_method1("delete", (key.as_str(),)) - .map_err(py_err)?; - Ok(()) - }) - } - - fn erase_prefix(&self, prefix: &StorePrefix) -> Result<(), StorageError> { - Python::attach(|py| { - self.0 - .bind(py) - .call_method1("delete_prefix", (prefix.as_str(),)) - .map_err(py_err)?; - Ok(()) - }) - } -} - -impl ListableStorageTraits for PyStore { - fn list(&self) -> Result { - Python::attach(|py| { - let keys: Vec = self - .0 - .bind(py) - .call_method0("list") - .map_err(py_err)? - .extract() - .map_err(py_err)?; - keys.into_iter() - .map(|k| StoreKey::new(k).map_err(invalid)) - .collect() - }) - } - - fn list_prefix(&self, prefix: &StorePrefix) -> Result { - Python::attach(|py| { - let keys: Vec = self - .0 - .bind(py) - .call_method1("list_prefix", (prefix.as_str(),)) - .map_err(py_err)? - .extract() - .map_err(py_err)?; - keys.into_iter() - .map(|k| StoreKey::new(k).map_err(invalid)) - .collect() - }) - } - - fn list_dir(&self, prefix: &StorePrefix) -> Result { - Python::attach(|py| { - let (keys, prefixes): (Vec, Vec) = self - .0 - .bind(py) - .call_method1("list_dir", (prefix.as_str(),)) - .map_err(py_err)? - .extract() - .map_err(py_err)?; - let keys = keys - .into_iter() - .map(|k| StoreKey::new(k).map_err(invalid)) - .collect::, StorageError>>()?; - let prefixes = prefixes - .into_iter() - .map(|p| StorePrefix::new(p).map_err(invalid)) - .collect::, StorageError>>()?; - Ok(StoreKeysPrefixes::new(keys, prefixes)) - }) - } - - fn size_prefix(&self, prefix: &StorePrefix) -> Result { - Python::attach(|py| { - self.0 - .bind(py) - .call_method1("getsize_prefix", (prefix.as_str(),)) - .map_err(py_err)? - .extract() - .map_err(py_err) - }) - } -} - -/// Like `resolve_store`, but also returns a cache key for the constructed -/// storage: `Some(root)` for native filesystem stores (which are safe to key an -/// Array cache on), `None` for the generic Python-callback path (uncached). -pub(crate) fn resolve_store_with_key( - obj: &Bound<'_, PyAny>, -) -> PyResult<(ReadableWritableListableStorage, Option)> { - if let Ok(config) = obj.cast::() { - if let Some(root) = config.get_item("filesystem")? { - let root: String = root.extract()?; - let store = - FilesystemStore::new(&root).map_err(|e| PyValueError::new_err(e.to_string()))?; - return Ok((Arc::new(store), Some(root))); - } - return Err(PyValueError::new_err("unrecognized store configuration")); - } - Ok((Arc::new(PyStore(obj.clone().unbind())), None)) -} - -/// Convert the Python-side store representation (`zarr.zarrs._bridge.resolve_store` -/// output) into a zarrs storage handle. -pub(crate) fn resolve_store(obj: &Bound<'_, PyAny>) -> PyResult { - Ok(resolve_store_with_key(obj)?.0) -} diff --git a/pyproject.toml b/pyproject.toml index 932387186a..4d930efe30 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -7,7 +7,6 @@ exclude = [ "/.github", "/bench", "/docs", - "/packages/zarrs-bindings", ] [project] @@ -135,10 +134,6 @@ dev = [ "universal-pathlib", "mypy", ] -zarrs = [ - {include-group = "test"}, - "zarrs-bindings", -] zarrista = [ {include-group = "test"}, "zarrista @ git+https://github.com/developmentseed/zarrista@95e47ad4c414c5920f0cf15550f923039641da8e", @@ -450,7 +445,6 @@ addopts = [ "--doctest-modules", "--ignore=tests/test_regression/scripts", "--ignore=src/zarr/_cli", - "--ignore=src/zarr/zarrs", ] filterwarnings = [ "error", @@ -506,6 +500,3 @@ ignore-words-list = "astroid" [project.entry-points.pytest11] zarr = "zarr.testing" - -[tool.uv.sources] -zarrs-bindings = { path = "packages/zarrs-bindings" } diff --git a/src/zarr/core/config.py b/src/zarr/core/config.py index 288f56de69..7dcbc78e31 100644 --- a/src/zarr/core/config.py +++ b/src/zarr/core/config.py @@ -107,7 +107,6 @@ def enable_gpu(self) -> ConfigSet: "path": "zarr.core.codec_pipeline.BatchedCodecPipeline", "batch_size": 1, }, - "crud": {"backend": "reference"}, "codecs": { "blosc": "zarr.codecs.blosc.BloscCodec", "gzip": "zarr.codecs.gzip.GzipCodec", diff --git a/src/zarr/crud/__init__.py b/src/zarr/crud/__init__.py deleted file mode 100644 index 33b01668b2..0000000000 --- a/src/zarr/crud/__init__.py +++ /dev/null @@ -1,55 +0,0 @@ -""" -Backend-agnostic low-level functional CRUD API for zarr hierarchies. - -The public functions delegate byte- and metadata-level work to a `CrudBackend`. -Two backends ship: a pure-Python reference backend (the default) and a -zarrs-accelerated backend (`zarr.zarrs`, requires the `zarrs-bindings` -extension). Select one with the `crud.backend` config key or a per-call -`backend=` argument. - -Array routines take an explicit metadata document (a `dict` matching the -`zarr.json` / `.zarray` document) rather than reading it from the store, which -makes read-only and virtual views possible. -""" - -from zarr.crud._api import ( - CrudOptions, - create_new_array, - create_new_group, - create_overwrite_array, - create_overwrite_group, - delete_chunk, - delete_node, - list_children, - read_chunk, - read_encoded_chunk, - read_metadata, - read_region, - write_chunk, -) -from zarr.crud._backend import CrudBackend, NodeExistsError -from zarr.crud._reference import ReferenceBackend -from zarr.crud._registry import get_backend, register_backend - -register_backend("reference", ReferenceBackend()) - -__all__ = [ - "CrudBackend", - "CrudOptions", - "NodeExistsError", - "ReferenceBackend", - "create_new_array", - "create_new_group", - "create_overwrite_array", - "create_overwrite_group", - "delete_chunk", - "delete_node", - "get_backend", - "list_children", - "read_chunk", - "read_encoded_chunk", - "read_metadata", - "read_region", - "register_backend", - "write_chunk", -] diff --git a/src/zarr/crud/_api.py b/src/zarr/crud/_api.py deleted file mode 100644 index 91aeef5007..0000000000 --- a/src/zarr/crud/_api.py +++ /dev/null @@ -1,344 +0,0 @@ -from __future__ import annotations - -import operator -import types -from collections.abc import Sequence -from dataclasses import dataclass -from typing import TYPE_CHECKING, Any, cast - -import numpy as np - -from zarr.core.buffer.core import default_buffer_prototype -from zarr.crud._common import parse_array_metadata -from zarr.crud._registry import get_backend - -if TYPE_CHECKING: - from collections.abc import Mapping - - import numpy.typing as npt - - from zarr.abc.store import Store - from zarr.core.common import JSON - from zarr.crud._backend import CrudBackend - - -@dataclass(frozen=True, slots=True) -class CrudOptions: - """Options for CRUD operations. - - Currently empty: fields (concurrency limits, checksum validation) arrive in - a later phase. Accepting it now keeps signatures stable. - """ - - -BasicIndex = int | slice | types.EllipsisType -BasicSelection = BasicIndex | tuple[BasicIndex, ...] - - -def _resolve_backend(backend: CrudBackend | str | None) -> CrudBackend: - if backend is None or isinstance(backend, str): - return get_backend(backend) - return backend - - -def _chunk_dtype_and_shape( - metadata: Mapping[str, JSON], -) -> tuple[np.dtype[Any], tuple[int, ...]]: - """Resolve native-byte-order numpy dtype and regular chunk shape. - - Backends decode to (and encode from) the native in-memory representation, - applying any byte-order codec themselves, so the dtype is coerced to native. - """ - from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata - - meta_obj = parse_array_metadata(metadata) - if isinstance(meta_obj, ArrayV3Metadata): - grid = meta_obj.chunk_grid - if not isinstance(grid, RegularChunkGridMetadata): - raise NotImplementedError("only regular chunk grids are supported") - chunk_shape = tuple(grid.chunk_shape) - else: - chunk_shape = tuple(meta_obj.chunks) - return meta_obj.dtype.to_native_dtype().newbyteorder("="), chunk_shape - - -def _array_shape(metadata: Mapping[str, JSON]) -> tuple[int, ...]: - shape = metadata.get("shape") - if not isinstance(shape, Sequence) or isinstance(shape, str): - raise TypeError("metadata document has no valid 'shape'") - result: list[int] = [] - for s in shape: - if not isinstance(s, (int, float)): - raise TypeError(f"shape element {s!r} is not a number") - if isinstance(s, float) and not s.is_integer(): - raise TypeError(f"shape element {s!r} is not an integer") - result.append(int(s)) - return tuple(result) - - -def _chunk_key(metadata: Mapping[str, JSON], path: str, coords: tuple[int, ...]) -> str: - meta_obj = parse_array_metadata(metadata) - rel = meta_obj.encode_chunk_key(coords) - p = path.strip("/") - return f"{p}/{rel}" if p else rel - - -def _normalize_selection( - selection: BasicSelection, shape: tuple[int, ...] -) -> tuple[list[int], list[int], tuple[slice | int, ...]]: - """Normalize a numpy basic-indexing selection to a step-1 bounding box. - - Returns `(start, bounding_shape, post_index)`: the box to fetch and the - numpy index to apply to it (strides, reversals, integer-axis removal). Only - integers, slices, and `Ellipsis` are supported; fancy indexing raises. - """ - sel_tuple = selection if isinstance(selection, tuple) else (selection,) - - n_ellipsis = sum(1 for s in sel_tuple if s is Ellipsis) - if n_ellipsis > 1: - raise IndexError("an index can only have a single ellipsis ('...')") - if n_ellipsis == 1: - i = sel_tuple.index(Ellipsis) - n_fill = len(shape) - (len(sel_tuple) - 1) - if n_fill < 0: - raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") - sel_tuple = sel_tuple[:i] + (slice(None),) * n_fill + sel_tuple[i + 1 :] - if len(sel_tuple) > len(shape): - raise IndexError(f"too many indices for array: array is {len(shape)}-dimensional") - sel_tuple = sel_tuple + (slice(None),) * (len(shape) - len(sel_tuple)) - - starts: list[int] = [] - lengths: list[int] = [] - post: list[slice | int] = [] - for dim, (sel, size) in enumerate(zip(sel_tuple, shape, strict=True)): - if isinstance(sel, slice): - start, stop, step = sel.indices(size) - n = len(range(start, stop, step)) - if n == 0: - starts.append(0) - lengths.append(0) - post.append(slice(None)) - elif step > 0: - last = start + (n - 1) * step - starts.append(start) - lengths.append(last - start + 1) - post.append(slice(None, None, step)) - else: - last = start + (n - 1) * step - starts.append(last) - lengths.append(start - last + 1) - post.append(slice(None, None, step)) - else: - assert not isinstance(sel, types.EllipsisType), "Ellipsis already expanded above" - try: - idx = operator.index(sel) - except TypeError: - raise TypeError( - "unsupported selection element " - f"{sel!r}: only integers, slices, and Ellipsis are supported" - ) from None - if idx < 0: - idx += size - if not 0 <= idx < size: - raise IndexError(f"index {sel} is out of bounds for axis {dim} with size {size}") - starts.append(idx) - lengths.append(1) - post.append(0) - return starts, lengths, tuple(post) - - -# --- node lifecycle --- - - -async def create_new_group( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Create a group from a group metadata document. Raises `NodeExistsError` - if a node already exists at `path`. Not atomic against concurrent writers.""" - await _resolve_backend(backend).create_group(store, path, metadata, overwrite=False) - - -async def create_overwrite_group( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Create a group, deleting any existing node (and children) first. Not - atomic against concurrent writers.""" - await _resolve_backend(backend).create_group(store, path, metadata, overwrite=True) - - -async def create_new_array( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Create an array from a v2 or v3 metadata document. Raises - `NodeExistsError` if a node already exists. Not atomic against concurrent - writers.""" - await _resolve_backend(backend).create_array(store, path, metadata, overwrite=False) - - -async def create_overwrite_array( - metadata: Mapping[str, JSON], - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Create an array, deleting any existing node (and children) first. Not - atomic against concurrent writers.""" - await _resolve_backend(backend).create_array(store, path, metadata, overwrite=True) - - -async def read_metadata( - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> dict[str, JSON]: - """Read the metadata document of the array or group at `path`. Raises - `zarr.errors.NodeNotFoundError` if no node exists there.""" - return await _resolve_backend(backend).read_metadata(store, path) - - -async def delete_node( - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Delete the node at `path` and everything under it. Raises - `zarr.errors.NodeNotFoundError` if absent. `path=""` clears the store.""" - await _resolve_backend(backend).delete_node(store, path) - - -async def list_children( - store: Store, - path: str, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> list[tuple[str, dict[str, JSON]]]: - """List the direct children of the group at `path` as - `(path, metadata_document)` pairs (store-relative, no leading `/`). Raises - `zarr.errors.NodeNotFoundError` if no group exists there.""" - return await _resolve_backend(backend).list_children(store, path) - - -# --- chunk I/O --- - - -async def read_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> np.ndarray[Any, np.dtype[Any]]: - """Read and decode the whole chunk at `chunk_coords`. The metadata document - is authoritative; missing chunks decode to the fill value. The result is a - read-only view (`.copy()` for a writable array).""" - be = _resolve_backend(backend) - raw = await be.read_chunk(store, path, metadata, tuple(chunk_coords)) - dtype, chunk_shape = _chunk_dtype_and_shape(metadata) - return np.frombuffer(raw, dtype=dtype).reshape(chunk_shape) - - -async def read_encoded_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> bytes | None: - """Read the raw, still-encoded bytes of the chunk at `chunk_coords`, or - `None` if absent. Pure store I/O (`store.get` on the chunk key): the - `backend` argument is accepted for signature uniformity but unused.""" - key = _chunk_key(metadata, path, tuple(chunk_coords)) - buf = await store.get(key, prototype=default_buffer_prototype()) - return None if buf is None else buf.to_bytes() - - -async def write_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - value: npt.ArrayLike, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Encode `value` with the codecs in `metadata` and store it as the chunk at - `chunk_coords`. `value` must match the chunk shape exactly.""" - be = _resolve_backend(backend) - dtype, chunk_shape = _chunk_dtype_and_shape(metadata) - arr = np.ascontiguousarray(np.asarray(value, dtype=dtype)) - if arr.shape != chunk_shape: - raise ValueError(f"value shape {arr.shape} does not match chunk shape {chunk_shape}") - await be.write_chunk(store, path, metadata, tuple(chunk_coords), arr.tobytes()) - - -async def delete_chunk( - metadata: Mapping[str, JSON], - store: Store, - path: str, - chunk_coords: tuple[int, ...], - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> None: - """Delete the chunk at `chunk_coords`. Deleting a missing chunk is a no-op.""" - await _resolve_backend(backend).delete_chunk(store, path, metadata, tuple(chunk_coords)) - - -# --- region I/O --- - - -async def read_region( - metadata: Mapping[str, JSON], - store: Store, - path: str, - selection: BasicSelection, - *, - options: CrudOptions | None = None, - backend: CrudBackend | str | None = None, -) -> np.ndarray[Any, np.dtype[Any]]: - """Read and decode a region given by a numpy basic-indexing `selection` - (integers, slices with steps, `Ellipsis`). One backend call fetches the - step-1 bounding box; strides/reversals/integer-axis removal are applied as - numpy views. Missing chunks decode to the fill value. Fancy indexing raises - `TypeError`. The result is a read-only view. - - Note: a `slice(0, N, step)` reads `O(N)` bytes even though `O(N / step)` are - returned; for sparse selections over large arrays prefer `read_chunk`.""" - be = _resolve_backend(backend) - dtype, _ = _chunk_dtype_and_shape(metadata) - shape = _array_shape(metadata) - starts, lengths, post_index = _normalize_selection(selection, shape) - if 0 in lengths: - block = np.empty(lengths, dtype=dtype) - block.flags.writeable = False - else: - raw = await be.read_subset(store, path, metadata, tuple(starts), tuple(lengths)) - block = np.frombuffer(raw, dtype=dtype).reshape(lengths) - return cast("np.ndarray[Any, np.dtype[Any]]", block[post_index]) diff --git a/src/zarr/crud/_backend.py b/src/zarr/crud/_backend.py deleted file mode 100644 index 638dacf6f6..0000000000 --- a/src/zarr/crud/_backend.py +++ /dev/null @@ -1,76 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING, Protocol, runtime_checkable - -if TYPE_CHECKING: - from collections.abc import Mapping, Sequence - - from zarr.abc.store import Store - from zarr.core.common import JSON - - -class NodeExistsError(ValueError): - """Raised when a node already exists at a path and overwrite was not requested.""" - - -@runtime_checkable -class CrudBackend(Protocol): - """The byte/metadata-level contract a CRUD backend must implement. - - Methods take neutral types: the metadata document as a `dict`, a zarr - `Store`, and plain zarr paths (`""`, `"foo/bar"`). They return raw bytes, - parsed JSON documents, or `None`. The shared `zarr.crud` facade builds the - numpy- and selection-level API on top of these. - - `create_*` raise `zarr.crud.NodeExistsError` when a node exists and - `overwrite` is false. `read_metadata`/`delete_node`/`list_children` raise - `zarr.errors.NodeNotFoundError` when the target is missing. - - Note: because this protocol is `runtime_checkable`, `isinstance` checks only - verify that the method names exist, not their signatures or that they are - async. Static type checking (mypy) is the authoritative conformance check. - - `read_chunk` and `read_subset` must return immutable `bytes` (not - `bytearray`): the facade wraps them with `numpy.frombuffer`, which yields a - read-only array only for immutable buffers. - """ - - async def create_array( - self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool - ) -> None: ... - - async def create_group( - self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool - ) -> None: ... - - async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: ... - - async def read_chunk( - self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] - ) -> bytes: ... - - async def read_subset( - self, - store: Store, - path: str, - metadata: Mapping[str, JSON], - start: Sequence[int], - shape: Sequence[int], - ) -> bytes: ... - - async def write_chunk( - self, - store: Store, - path: str, - metadata: Mapping[str, JSON], - coords: tuple[int, ...], - data: bytes, - ) -> None: ... - - async def delete_chunk( - self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] - ) -> None: ... - - async def delete_node(self, store: Store, path: str) -> None: ... - - async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: ... diff --git a/src/zarr/crud/_common.py b/src/zarr/crud/_common.py deleted file mode 100644 index 4837edfa03..0000000000 --- a/src/zarr/crud/_common.py +++ /dev/null @@ -1,21 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING - -from zarr.core.metadata.v2 import ArrayV2Metadata -from zarr.core.metadata.v3 import ArrayV3Metadata - -if TYPE_CHECKING: - from collections.abc import Mapping - - from zarr.core.common import JSON - - -def parse_array_metadata( - metadata: Mapping[str, JSON], -) -> ArrayV3Metadata | ArrayV2Metadata: - """Parse a metadata document into a v2 or v3 array metadata object.""" - data = dict(metadata) - if data.get("zarr_format") == 3: - return ArrayV3Metadata.from_dict(data) - return ArrayV2Metadata.from_dict(data) diff --git a/src/zarr/crud/_reference.py b/src/zarr/crud/_reference.py deleted file mode 100644 index 78db4c6e04..0000000000 --- a/src/zarr/crud/_reference.py +++ /dev/null @@ -1,203 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING, Any - -import numpy as np - -from zarr.core.array import AsyncArray, create_codec_pipeline -from zarr.core.array_spec import ArrayConfig, ArraySpec -from zarr.core.buffer.core import NDBuffer, default_buffer_prototype -from zarr.core.common import ZARR_JSON, ZARRAY_JSON, ZATTRS_JSON, ZGROUP_JSON -from zarr.core.group import GroupMetadata -from zarr.core.metadata.io import save_metadata -from zarr.core.metadata.v2 import ArrayV2Metadata -from zarr.core.metadata.v3 import ArrayV3Metadata, RegularChunkGridMetadata -from zarr.crud._backend import NodeExistsError -from zarr.crud._common import parse_array_metadata -from zarr.errors import NodeNotFoundError -from zarr.storage._common import StorePath - -if TYPE_CHECKING: - from collections.abc import Mapping, Sequence - - from zarr.abc.store import Store - from zarr.core.common import JSON - - -def _native_dtype(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> np.dtype[Any]: - """Numpy dtype in native byte order (zarrs and the facade assume native).""" - return meta_obj.dtype.to_native_dtype().newbyteorder("=") - - -def _chunk_shape(meta_obj: ArrayV3Metadata | ArrayV2Metadata) -> tuple[int, ...]: - if isinstance(meta_obj, ArrayV3Metadata): - grid = meta_obj.chunk_grid - if not isinstance(grid, RegularChunkGridMetadata): - raise NotImplementedError("only regular chunk grids are supported") - return tuple(grid.chunk_shape) - return tuple(meta_obj.chunks) - - -def _array_spec(meta_obj: ArrayV3Metadata | ArrayV2Metadata, shape: tuple[int, ...]) -> ArraySpec: - order = meta_obj.order if isinstance(meta_obj, ArrayV2Metadata) else "C" - return ArraySpec( - shape=shape, - dtype=meta_obj.dtype, - fill_value=meta_obj.fill_value, - config=ArrayConfig.from_dict({"order": order}), - prototype=default_buffer_prototype(), - ) - - -def _is_all_fill_value( - arr: np.ndarray[Any, np.dtype[Any]], fill_value: Any, dtype: np.dtype[Any] -) -> bool: - """Whether every element of `arr` equals the fill value (NaN-aware for floats).""" - if fill_value is None: - return False - fill = np.asarray(fill_value, dtype=dtype) - if np.issubdtype(dtype, np.floating) or np.issubdtype(dtype, np.complexfloating): - return bool(np.array_equal(arr, np.broadcast_to(fill, arr.shape), equal_nan=True)) - return bool(np.all(arr == fill)) - - -class ReferenceBackend: - """Pure-Python CRUD backend wrapping zarr-python's own machinery. - - Constructs no high-level `Array` for chunk operations (it drives the codec - pipeline directly); it does reuse `AsyncArray.getitem` for multi-chunk - subset reads, which is exactly the `BasicIndexer` + codec-pipeline read path. - """ - - async def _node_exists(self, store: Store, path: str) -> bool: - proto = default_buffer_prototype() - sp = StorePath(store, path.strip("/")) - for meta_key in (ZARR_JSON, ZARRAY_JSON, ZGROUP_JSON): - if await (sp / meta_key).get(prototype=proto) is not None: - return True - return False - - async def create_array( - self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool - ) -> None: - meta_obj = parse_array_metadata(metadata) - await self._create(store, path, meta_obj, overwrite=overwrite) - - async def create_group( - self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool - ) -> None: - meta_obj = GroupMetadata.from_dict(dict(metadata)) - await self._create(store, path, meta_obj, overwrite=overwrite) - - async def _create(self, store: Store, path: str, meta_obj: Any, *, overwrite: bool) -> None: - sp = StorePath(store, path.strip("/")) - if overwrite: - await store.delete_dir(path.strip("/")) - elif await self._node_exists(store, path): - raise NodeExistsError(f"a node already exists at path {path!r}") - await save_metadata(sp, meta_obj, ensure_parents=True) - - async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: - from zarr.core._json import buffer_to_json_object - - proto = default_buffer_prototype() - sp = StorePath(store, path.strip("/")) - buf = await (sp / ZARR_JSON).get(prototype=proto) - if buf is not None: - return buffer_to_json_object(buf) - for meta_key in (ZARRAY_JSON, ZGROUP_JSON): - b = await (sp / meta_key).get(prototype=proto) - if b is not None: - doc = buffer_to_json_object(b) - zattrs = await (sp / ZATTRS_JSON).get(prototype=proto) - if zattrs is not None: - doc["attributes"] = buffer_to_json_object(zattrs) - return doc - raise NodeNotFoundError(f"no node found at path {path!r}") - - async def read_chunk( - self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] - ) -> bytes: - meta_obj = parse_array_metadata(metadata) - shape = _chunk_shape(meta_obj) - np_dtype = _native_dtype(meta_obj) - sp = StorePath(store, path.strip("/")) - chunk_key = meta_obj.encode_chunk_key(coords) - buf = await (sp / chunk_key).get(prototype=default_buffer_prototype()) - if buf is None: - arr = np.full(shape, meta_obj.fill_value, dtype=np_dtype) - else: - pipeline = create_codec_pipeline(meta_obj) - spec = _array_spec(meta_obj, shape) - decoded = list(await pipeline.decode([(buf, spec)])) - nd_buf = decoded[0] - if nd_buf is None: - arr = np.full(shape, meta_obj.fill_value, dtype=np_dtype) - else: - arr = np.asarray(nd_buf.as_numpy_array(), dtype=np_dtype) - return np.ascontiguousarray(arr).tobytes() - - async def read_subset( - self, - store: Store, - path: str, - metadata: Mapping[str, JSON], - start: Sequence[int], - shape: Sequence[int], - ) -> bytes: - meta_obj = parse_array_metadata(metadata) - np_dtype = _native_dtype(meta_obj) - async_arr = AsyncArray(metadata=meta_obj, store_path=StorePath(store, path.strip("/"))) - selection = tuple(slice(s, s + length) for s, length in zip(start, shape, strict=True)) - result = await async_arr.getitem(selection) - return np.ascontiguousarray(np.asarray(result, dtype=np_dtype)).tobytes() - - async def write_chunk( - self, - store: Store, - path: str, - metadata: Mapping[str, JSON], - coords: tuple[int, ...], - data: bytes, - ) -> None: - meta_obj = parse_array_metadata(metadata) - shape = _chunk_shape(meta_obj) - np_dtype = _native_dtype(meta_obj) - sp = StorePath(store, path.strip("/")) - chunk_key = meta_obj.encode_chunk_key(coords) - arr = np.frombuffer(data, dtype=np_dtype).reshape(shape) - if _is_all_fill_value(arr, meta_obj.fill_value, np_dtype): - await (sp / chunk_key).delete() - return - pipeline = create_codec_pipeline(meta_obj) - spec = _array_spec(meta_obj, shape) - encoded = list(await pipeline.encode([(NDBuffer.from_ndarray_like(arr), spec)])) - buf = encoded[0] - if buf is None: - await (sp / chunk_key).delete() - else: - await (sp / chunk_key).set(buf) - - async def delete_chunk( - self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] - ) -> None: - meta_obj = parse_array_metadata(metadata) - sp = StorePath(store, path.strip("/")) - await (sp / meta_obj.encode_chunk_key(coords)).delete() - - async def delete_node(self, store: Store, path: str) -> None: - if not await self._node_exists(store, path): - raise NodeNotFoundError(f"no node found at path {path!r}") - await store.delete_dir(path.strip("/")) - - async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: - p = path.strip("/") - if not await self._node_exists(store, path): - raise NodeNotFoundError(f"no node found at path {path!r}") - prefix = f"{p}/" if p else "" - children: list[tuple[str, dict[str, JSON]]] = [] - async for name in store.list_dir(prefix): - child_path = f"{p}/{name}" if p else name - if await self._node_exists(store, child_path): - children.append((name, await self.read_metadata(store, child_path))) - return children diff --git a/src/zarr/crud/_registry.py b/src/zarr/crud/_registry.py deleted file mode 100644 index 84fde1bc20..0000000000 --- a/src/zarr/crud/_registry.py +++ /dev/null @@ -1,40 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING - -from zarr.core.config import config - -if TYPE_CHECKING: - from zarr.crud._backend import CrudBackend - -# Backends are registered at import time (reference by zarr.crud, zarrs by -# zarr.zarrs). CPython's import lock plus the GIL make this dict safe without -# additional locking. -_BACKENDS: dict[str, CrudBackend] = {} - - -def register_backend(name: str, backend: CrudBackend) -> None: - """Register a CRUD backend instance under `name`.""" - _BACKENDS[name] = backend - - -def get_backend(name: str | None = None) -> CrudBackend: - """Resolve a backend by name, or the configured default when `name` is None. - - Selecting `"zarrs"` imports `zarr.zarrs` if needed so it can self-register. - """ - if name is None: - name = config.get("crud.backend") - if name not in _BACKENDS and name == "zarrs": - # "reference" is pre-registered by zarr.crud at import; "zarrs" lives in a - # separate package that may not be imported yet, so load it on demand. - try: - import zarr.zarrs # noqa: F401 (import registers the zarrs backend) - except ImportError as e: - raise ImportError( - "the 'zarrs' CRUD backend requires the zarrs-bindings extension; " - "install it with: uv sync --group zarrs" - ) from e - if name not in _BACKENDS: - raise KeyError(f"no CRUD backend registered as {name!r}; registered: {sorted(_BACKENDS)}") - return _BACKENDS[name] diff --git a/src/zarr/zarrs/__init__.py b/src/zarr/zarrs/__init__.py deleted file mode 100644 index bff68ade62..0000000000 --- a/src/zarr/zarrs/__init__.py +++ /dev/null @@ -1,26 +0,0 @@ -""" -The zarrs CRUD backend for `zarr.crud`, backed by the Rust -[`zarrs`](https://zarrs.dev) crate. - -Importing this module registers the `"zarrs"` backend. Requires the -`zarrs-bindings` extension (in-repo Rust crate; `uv sync --group zarrs`). Select -it with `zarr.config.set({"crud.backend": "zarrs"})` or per call via -`backend="zarrs"`. -""" - -try: - import _zarrs_bindings -except ImportError as e: - raise ImportError( - "zarr.zarrs requires the `zarrs-bindings` package, which is not installed. " - "It is built from the zarr-python repository: run `uv sync --group zarrs`." - ) from e - -from zarr.crud import register_backend -from zarr.zarrs._backend import ZarrsBackend - -__version__: str = _zarrs_bindings.version() - -register_backend("zarrs", ZarrsBackend()) - -__all__ = ["ZarrsBackend", "__version__"] diff --git a/src/zarr/zarrs/_backend.py b/src/zarr/zarrs/_backend.py deleted file mode 100644 index e95759660b..0000000000 --- a/src/zarr/zarrs/_backend.py +++ /dev/null @@ -1,150 +0,0 @@ -from __future__ import annotations - -import asyncio -import json -from contextlib import contextmanager -from typing import TYPE_CHECKING, cast - -import _zarrs_bindings as _zb - -from zarr.crud import NodeExistsError -from zarr.errors import NodeNotFoundError -from zarr.zarrs._bridge import resolve_store - -if TYPE_CHECKING: - from collections.abc import Iterator, Mapping, Sequence - - from zarr.abc.store import Store - from zarr.core.common import JSON - - -def _node_path(path: str) -> str: - """Convert a zarr path (`""`, `"foo/bar"`) to a zarrs node path (`"/"`, - `"/foo/bar"`).""" - return f"/{path.strip('/')}" - - -@contextmanager -def _translate_errors() -> Iterator[None]: - try: - yield - except _zb.NodeNotFoundError as err: - raise NodeNotFoundError(str(err)) from err - except _zb.NodeExistsError as err: - raise NodeExistsError(str(err)) from err - - -class ZarrsBackend: - """CRUD backend backed by the Rust `zarrs` crate via `_zarrs_bindings`. - - Owns the zarrs-specific plumbing: JSON-serializing the metadata document, - the `/`-prefixed node-path form, store resolution, offloading the blocking - Rust calls to a worker thread, and translating binding exceptions to the - canonical `zarr.crud` / `zarr.errors` types. - - Known limitation: creating a Zarr v2 *group* with attributes writes a - non-standard `.zattrs` (the attributes nested under an ``"attributes"`` key) - that zarr-python and other readers interpret incorrectly. This is a - zarrs-crate behavior; the pure-Python reference backend writes the standard - layout. Prefer the reference backend for writing v2 groups until the zarrs - crate is fixed. - """ - - async def create_array( - self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool - ) -> None: - with _translate_errors(): - await asyncio.to_thread( - _zb.create_array, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - overwrite, - ) - - async def create_group( - self, store: Store, path: str, metadata: Mapping[str, JSON], *, overwrite: bool - ) -> None: - with _translate_errors(): - await asyncio.to_thread( - _zb.create_group, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - overwrite, - ) - - async def read_metadata(self, store: Store, path: str) -> dict[str, JSON]: - with _translate_errors(): - raw = await asyncio.to_thread(_zb.read_metadata, resolve_store(store), _node_path(path)) - return cast("dict[str, JSON]", json.loads(raw)) - - async def read_chunk( - self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] - ) -> bytes: - return await asyncio.to_thread( - _zb.retrieve_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(coords), - ) - - async def read_subset( - self, - store: Store, - path: str, - metadata: Mapping[str, JSON], - start: Sequence[int], - shape: Sequence[int], - ) -> bytes: - return await asyncio.to_thread( - _zb.retrieve_array_subset, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(start), - list(shape), - ) - - async def write_chunk( - self, - store: Store, - path: str, - metadata: Mapping[str, JSON], - coords: tuple[int, ...], - data: bytes, - ) -> None: - await asyncio.to_thread( - _zb.store_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(coords), - data, - ) - - async def delete_chunk( - self, store: Store, path: str, metadata: Mapping[str, JSON], coords: tuple[int, ...] - ) -> None: - await asyncio.to_thread( - _zb.erase_chunk, - resolve_store(store), - _node_path(path), - json.dumps(metadata), - list(coords), - ) - - async def delete_node(self, store: Store, path: str) -> None: - with _translate_errors(): - await asyncio.to_thread(_zb.delete_node, resolve_store(store), _node_path(path)) - - async def list_children(self, store: Store, path: str) -> list[tuple[str, dict[str, JSON]]]: - with _translate_errors(): - raw: list[tuple[str, str]] = await asyncio.to_thread( - _zb.list_children, resolve_store(store), _node_path(path) - ) - return [ - (child_path.lstrip("/"), cast("dict[str, JSON]", json.loads(doc))) - for child_path, doc in raw - ] diff --git a/src/zarr/zarrs/_bridge.py b/src/zarr/zarrs/_bridge.py deleted file mode 100644 index e7632647ad..0000000000 --- a/src/zarr/zarrs/_bridge.py +++ /dev/null @@ -1,105 +0,0 @@ -from __future__ import annotations - -import builtins -from typing import TYPE_CHECKING - -from zarr.abc.store import OffsetByteRequest, RangeByteRequest, SuffixByteRequest -from zarr.core.buffer.core import default_buffer_prototype -from zarr.core.sync import _collect_aiterator, sync -from zarr.storage import LocalStore - -if TYPE_CHECKING: - from zarr.abc.store import Store - -# Alias to avoid shadowing the `list` builtin with the `StoreShim.list` method -# in mypy's class-scope name resolution. -_list = builtins.list - - -class StoreShim: - """ - Synchronous adapter over an async `Store`, called from Rust worker threads. - - Each method blocks the calling thread by submitting a coroutine to the zarr - event-loop thread (`zarr.core.sync`). Methods must never be called from the - zarr event-loop thread itself; the Rust bindings only call them from - `asyncio.to_thread` worker threads. - """ - - def __init__(self, store: Store) -> None: - self._store = store - self._prototype = default_buffer_prototype() - - def get(self, key: str) -> bytes | None: - buf = sync(self._store.get(key, prototype=self._prototype)) - return None if buf is None else buf.to_bytes() - - def get_range(self, key: str, offset: int, length: int | None) -> bytes | None: - byte_range = ( - RangeByteRequest(offset, offset + length) - if length is not None - else OffsetByteRequest(offset) - ) - buf = sync(self._store.get(key, prototype=self._prototype, byte_range=byte_range)) - return None if buf is None else buf.to_bytes() - - def get_suffix(self, key: str, suffix: int) -> bytes | None: - buf = sync( - self._store.get(key, prototype=self._prototype, byte_range=SuffixByteRequest(suffix)) - ) - return None if buf is None else buf.to_bytes() - - def set(self, key: str, value: bytes) -> None: - sync(self._store.set(key, self._prototype.buffer.from_bytes(value))) - - def delete(self, key: str) -> None: - sync(self._store.delete(key)) - - def delete_prefix(self, prefix: str) -> None: - sync(self._store.delete_dir(prefix.rstrip("/"))) - - def getsize(self, key: str) -> int | None: - try: - return sync(self._store.getsize(key)) - except FileNotFoundError: - return None - - def getsize_prefix(self, prefix: str) -> int: - return sync(self._store.getsize_prefix(prefix.rstrip("/"))) - - def list(self) -> _list[str]: - return sorted(sync(_collect_aiterator(self._store.list()))) - - def list_prefix(self, prefix: str) -> _list[str]: - return sorted(sync(_collect_aiterator(self._store.list_prefix(prefix)))) - - def list_dir(self, prefix: str) -> tuple[_list[str], _list[str]]: - """Return `(keys, prefixes)` directly under `prefix`, as zarrs expects: - full keys, and child prefixes ending in `/`.""" - stripped = prefix.rstrip("/") - children = sorted(sync(_collect_aiterator(self._store.list_dir(stripped)))) - keys: _list[str] = [] - prefixes: _list[str] = [] - # A child is classified as a key iff it exists as one. Zarr hierarchies - # never store a bare key alongside same-named subkeys (e.g. "a" and - # "a/b"), so a name is never both a key and a prefix. - # TODO: replace the per-child exists() round-trip with a single listing - # pass when this becomes a bottleneck (remote stores). - for child in children: - full = f"{stripped}/{child}" if stripped else child - if sync(self._store.exists(full)): - keys.append(full) - else: - prefixes.append(full + "/") - return keys, prefixes - - -def resolve_store(store: Store) -> StoreShim | dict[str, str]: - """ - Convert a zarr `Store` into the representation `_zarrs_bindings` expects: - a config dict for stores with a native Rust implementation, otherwise a - `StoreShim` that Rust calls back into. - """ - if isinstance(store, LocalStore) and not store.read_only: - return {"filesystem": str(store.root)} - return StoreShim(store) diff --git a/tests/crud/__init__.py b/tests/crud/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/tests/crud/conftest.py b/tests/crud/conftest.py deleted file mode 100644 index fbf2cf9e02..0000000000 --- a/tests/crud/conftest.py +++ /dev/null @@ -1,95 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING, Any - -import numpy as np -import pytest - -import zarr -from zarr.storage import LocalStore, MemoryStore - -if TYPE_CHECKING: - from collections.abc import AsyncIterator - from pathlib import Path - - from zarr.abc.store import Store - - -def _zarrs_available() -> bool: - """Return True only if the zarrs CrudBackend is fully usable (registered).""" - try: - import _zarrs_bindings # noqa: F401 - except ImportError: - return False - try: - import zarr.zarrs - except ImportError: - return False - # The module might exist but not yet register the zarrs CrudBackend (e.g. - # Task 4 not yet merged). Verify registration before enabling the param. - try: - import zarr.crud - - zarr.crud.get_backend("zarrs") - except (ImportError, KeyError): - return False - return True - - -@pytest.fixture( - params=[ - "reference", - pytest.param( - "zarrs", - marks=pytest.mark.skipif( - not _zarrs_available(), reason="zarrs-bindings is not installed" - ), - ), - ] -) -def backend(request: pytest.FixtureRequest) -> str: - """A CRUD backend name. The zarrs param is skipped when the extension is absent.""" - import zarr.crud - - if request.param == "zarrs": - import zarr.zarrs # noqa: F401 (registers the zarrs backend) - return str(request.param) - - -@pytest.fixture(params=["memory", "local"]) -async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncIterator[Store]: - if request.param == "memory": - s: Store = await MemoryStore.open() - else: - s = await LocalStore.open(root=tmp_path / "store") - try: - yield s - finally: - s.close() - - -def array_metadata(**kwargs: Any) -> dict[str, Any]: - """An array metadata document built via zarr-python itself.""" - params: dict[str, Any] = { - "shape": (8, 8), - "chunks": (4, 4), - "dtype": "uint16", - "zarr_format": 3, - } | kwargs - arr = zarr.create_array(store=MemoryStore(), **params) - doc = dict(arr.metadata.to_dict()) - if params["zarr_format"] == 2: - doc.pop("attributes", None) - return doc - - -def filled(store: Store, **kwargs: Any) -> tuple[np.ndarray[Any, np.dtype[Any]], dict[str, Any]]: - """Create an 8x8 array 'a', fill it with a ramp, return (data, metadata).""" - params: dict[str, Any] = {"shape": (8, 8), "chunks": (4, 4), "dtype": "uint16"} | kwargs - arr = zarr.create_array(store=store, name="a", **params) - data = np.arange(64, dtype=params["dtype"]).reshape(8, 8) - arr[:, :] = data - doc = dict(arr.metadata.to_dict()) - if params.get("zarr_format") == 2: - doc.pop("attributes", None) - return data, doc diff --git a/tests/crud/test_crud.py b/tests/crud/test_crud.py deleted file mode 100644 index d4aa79e334..0000000000 --- a/tests/crud/test_crud.py +++ /dev/null @@ -1,319 +0,0 @@ -from __future__ import annotations - -import copy -import json -from typing import TYPE_CHECKING, Any - -import numpy as np -import pytest - -import zarr -from tests.crud.conftest import array_metadata, filled -from zarr.codecs import BloscCodec, GzipCodec, ZstdCodec -from zarr.core.buffer.core import default_buffer_prototype -from zarr.crud import ( - NodeExistsError, - create_new_array, - create_new_group, - create_overwrite_array, - create_overwrite_group, - delete_chunk, - delete_node, - list_children, - read_chunk, - read_encoded_chunk, - read_metadata, - read_region, - write_chunk, -) -from zarr.errors import NodeNotFoundError - -if TYPE_CHECKING: - from zarr.abc.store import Store - -GROUP_META: dict[str, Any] = {"zarr_format": 3, "node_type": "group", "attributes": {"answer": 42}} -GROUP_META_V2: dict[str, Any] = {"zarr_format": 2, "attributes": {"answer": 42}} - - -# --- node lifecycle --- - - -async def test_create_new_group(backend: str, store: Store) -> None: - await create_new_group(GROUP_META, store, "foo", backend=backend) - assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} - - -async def test_v2_group_attrs_zarr_python_compatible_reference(store: Store) -> None: - # The reference backend writes standard v2 `.zattrs` (the bare attributes - # dict), so zarr-python and other readers see the right attributes. - await create_new_group(GROUP_META_V2, store, "g2", backend="reference") - assert dict(zarr.open_group(store=store, path="g2", mode="r").attrs) == {"answer": 42} - - -@pytest.mark.xfail( - reason="the zarrs backend writes v2 group attributes in a non-standard `.zattrs` " - "layout (nested under an 'attributes' key) that zarr-python reads back wrong; " - "tracked zarrs-crate limitation", - strict=True, -) -async def test_v2_group_attrs_zarr_python_compatible_zarrs(store: Store) -> None: - pytest.importorskip("_zarrs_bindings", reason="zarrs-bindings is not installed") - import zarr.zarrs - - await create_new_group(GROUP_META_V2, store, "g2", backend="zarrs") - assert dict(zarr.open_group(store=store, path="g2", mode="r").attrs) == {"answer": 42} - - -async def test_create_new_group_existing_raises(backend: str, store: Store) -> None: - await create_new_group(GROUP_META, store, "foo", backend=backend) - with pytest.raises(NodeExistsError): - await create_new_group(GROUP_META, store, "foo", backend=backend) - - -async def test_create_overwrite_group_replaces_array(backend: str, store: Store) -> None: - arr = zarr.create_array(store=store, name="foo", shape=(4,), chunks=(2,), dtype="uint8") - arr[:] = 1 - await create_overwrite_group(GROUP_META, store, "foo", backend=backend) - assert dict(zarr.open_group(store=store, path="foo", mode="r").attrs) == {"answer": 42} - assert not await store.exists("foo/c/0") - - -async def test_create_new_array(backend: str, store: Store) -> None: - await create_new_array(array_metadata(), store, "arr", backend=backend) - a = zarr.open_array(store=store, path="arr", mode="r") - assert a.shape == (8, 8) - assert a.dtype == np.dtype("uint16") - - -async def test_create_new_array_v2(backend: str, store: Store) -> None: - await create_new_array(array_metadata(zarr_format=2), store, "arr", backend=backend) - assert zarr.open_array(store=store, path="arr", mode="r").metadata.zarr_format == 2 - - -async def test_create_overwrite_array(backend: str, store: Store) -> None: - zarr.create_group(store=store, path="arr") - await create_overwrite_array(array_metadata(), store, "arr", backend=backend) - assert zarr.open_array(store=store, path="arr", mode="r").shape == (8, 8) - - -async def test_read_metadata(backend: str, store: Store) -> None: - await create_new_array(array_metadata(), store, "arr", backend=backend) - observed = await read_metadata(store, "arr", backend=backend) - raw = await store.get("arr/zarr.json", prototype=default_buffer_prototype()) - assert raw is not None - assert observed == json.loads(raw.to_bytes()) - - -async def test_read_metadata_missing(backend: str, store: Store) -> None: - with pytest.raises(NodeNotFoundError): - await read_metadata(store, "nope", backend=backend) - - -async def test_delete_node(backend: str, store: Store) -> None: - arr = zarr.create_array(store=store, name="doomed", shape=(4,), chunks=(2,), dtype="uint8") - arr[:] = 1 - await delete_node(store, "doomed", backend=backend) - assert not await store.exists("doomed/zarr.json") - assert not await store.exists("doomed/c/0") - - -async def test_delete_node_missing(backend: str, store: Store) -> None: - with pytest.raises(NodeNotFoundError): - await delete_node(store, "nope", backend=backend) - - -async def test_list_children(backend: str, store: Store) -> None: - root = zarr.create_group(store=store) - root.create_group("sub_group", attributes={"kind": "group"}) - root.create_array("sub_array", shape=(4,), chunks=(2,), dtype="uint8") - by_path = dict(await list_children(store, "", backend=backend)) - assert set(by_path) == {"sub_group", "sub_array"} - assert by_path["sub_group"]["node_type"] == "group" - assert by_path["sub_array"]["node_type"] == "array" - assert not any(p.startswith("/") for p in by_path) - - -async def test_create_read_delete_v2_group(backend: str, store: Store) -> None: - await create_new_group(GROUP_META_V2, store, "g2", backend=backend) - meta = await read_metadata(store, "g2", backend=backend) - assert meta["zarr_format"] == 2 - with pytest.raises(NodeExistsError): - await create_new_group(GROUP_META_V2, store, "g2", backend=backend) - await delete_node(store, "g2", backend=backend) - with pytest.raises(NodeNotFoundError): - await read_metadata(store, "g2", backend=backend) - - -async def test_read_metadata_v2_array(backend: str, store: Store) -> None: - await create_new_array(array_metadata(zarr_format=2), store, "arr", backend=backend) - meta = await read_metadata(store, "arr", backend=backend) - assert meta["zarr_format"] == 2 - - -# --- chunk I/O --- - - -@pytest.mark.parametrize("dtype", ["uint8", "int32", "float64", "u2"]) -async def test_read_chunk_differential(backend: str, store: Store, dtype: str) -> None: - data, meta = filled(store, dtype=dtype) - observed = await read_chunk(meta, store, "a", (1, 0), backend=backend) - np.testing.assert_array_equal(observed, data[4:8, 0:4]) - - -@pytest.mark.parametrize( - "compressors", [None, (GzipCodec(),), (ZstdCodec(),), (BloscCodec(cname="lz4"),)] -) -async def test_read_chunk_codecs(backend: str, store: Store, compressors: Any) -> None: - data, meta = filled(store, compressors=compressors) - observed = await read_chunk(meta, store, "a", (0, 1), backend=backend) - np.testing.assert_array_equal(observed, data[0:4, 4:8]) - - -async def test_read_chunk_v2(backend: str, store: Store) -> None: - data, meta = filled(store, dtype=" None: - data, meta = filled(store, dtype="uint16", zarr_format=2, order="F") - observed = await read_chunk(meta, store, "a", (1, 1), backend=backend) - np.testing.assert_array_equal(observed, data[4:8, 4:8]) - - -async def test_read_chunk_sharding(backend: str, store: Store) -> None: - data, meta = filled(store, chunks=(2, 2), shards=(4, 4)) - observed = await read_chunk(meta, store, "a", (1, 1), backend=backend) - np.testing.assert_array_equal(observed, data[4:8, 4:8]) - - -async def test_read_chunk_missing_is_fill(backend: str, store: Store) -> None: - arr = zarr.create_array( - store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=7 - ) - meta = dict(arr.metadata.to_dict()) - observed = await read_chunk(meta, store, "a", (0, 0), backend=backend) - np.testing.assert_array_equal(observed, np.full((4, 4), 7, dtype="uint16")) - - -async def test_read_chunk_metadata_view(backend: str, store: Store) -> None: - data, meta = filled(store, dtype="uint16", compressors=None) - view = copy.deepcopy(meta) - view["data_type"] = "uint8" - view["shape"] = [8, 16] - view["chunk_grid"]["configuration"]["chunk_shape"] = [4, 8] - observed = await read_chunk(view, store, "a", (1, 0), backend=backend) - np.testing.assert_array_equal(observed, data[4:8, 0:4].view("uint8")) - - -async def test_read_chunk_readonly(backend: str, store: Store) -> None: - _, meta = filled(store) - observed = await read_chunk(meta, store, "a", (0, 0), backend=backend) - assert not observed.flags.writeable - - -async def test_write_chunk_differential(backend: str, store: Store) -> None: - meta = array_metadata() - await create_new_array(meta, store, "a", backend=backend) - value = np.arange(16, dtype="uint16").reshape(4, 4) - await write_chunk(meta, store, "a", (0, 1), value, backend=backend) - np.testing.assert_array_equal(zarr.open_array(store=store, path="a", mode="r")[0:4, 4:8], value) - - -async def test_write_chunk_shape_mismatch(backend: str, store: Store) -> None: - meta = array_metadata() - await create_new_array(meta, store, "a", backend=backend) - with pytest.raises(ValueError, match="chunk shape"): - await write_chunk( - meta, store, "a", (0, 0), np.zeros((2, 2), dtype="uint16"), backend=backend - ) - - -async def test_delete_chunk(backend: str, store: Store) -> None: - _data, meta = filled(store) - assert await store.exists("a/c/0/0") - await delete_chunk(meta, store, "a", (0, 0), backend=backend) - assert not await store.exists("a/c/0/0") - - -async def test_write_all_fill_chunk_is_dropped(backend: str, store: Store) -> None: - arr = zarr.create_array( - store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16", fill_value=0 - ) - meta = dict(arr.metadata.to_dict()) - await write_chunk(meta, store, "a", (0, 0), np.zeros((4, 4), dtype="uint16"), backend=backend) - assert not await store.exists("a/c/0/0") - np.testing.assert_array_equal( - await read_chunk(meta, store, "a", (0, 0), backend=backend), - np.zeros((4, 4), dtype="uint16"), - ) - - -async def test_overwrite_chunk_with_fill_removes_it(backend: str, store: Store) -> None: - _data, meta = filled(store) # chunk (0,0) exists with nonzero data, fill_value default 0 - assert await store.exists("a/c/0/0") - await write_chunk(meta, store, "a", (0, 0), np.zeros((4, 4), dtype="uint16"), backend=backend) - assert not await store.exists("a/c/0/0") - - -async def test_read_encoded_chunk_matches_store(backend: str, store: Store) -> None: - _, meta = filled(store) - raw = await read_encoded_chunk(meta, store, "a", (0, 0), backend=backend) - expected = await store.get("a/c/0/0", prototype=default_buffer_prototype()) - assert expected is not None - assert raw == expected.to_bytes() - - -async def test_read_encoded_chunk_missing_is_none(backend: str, store: Store) -> None: - arr = zarr.create_array(store=store, name="e", shape=(8, 8), chunks=(4, 4), dtype="uint16") - meta = dict(arr.metadata.to_dict()) - assert await read_encoded_chunk(meta, store, "e", (0, 0), backend=backend) is None - - -# --- region I/O --- - -SELECTIONS: list[Any] = [ - (slice(None), slice(None)), - (slice(2, 7), slice(1, 5)), - (slice(None), 3), - (5, slice(None)), - (3, 4), - (slice(1, 8, 2), slice(None)), - (slice(None), slice(6, 1, -2)), - (slice(-3, None), slice(None, -1)), - ..., - (..., slice(2, 4)), - (slice(0, 0), slice(None)), - (slice(2, 6),), -] - - -@pytest.mark.parametrize("sel", SELECTIONS) -async def test_read_region_differential(backend: str, store: Store, sel: Any) -> None: - data, meta = filled(store) - observed = await read_region(meta, store, "a", sel, backend=backend) - np.testing.assert_array_equal(observed, data[sel]) - - -async def test_read_region_sharding(backend: str, store: Store) -> None: - data, meta = filled(store, chunks=(2, 2), shards=(4, 4)) - observed = await read_region(meta, store, "a", (slice(1, 7), slice(3, 8)), backend=backend) - np.testing.assert_array_equal(observed, data[1:7, 3:8]) - - -async def test_read_region_too_many_indices(backend: str, store: Store) -> None: - _, meta = filled(store) - with pytest.raises(IndexError, match="too many indices"): - await read_region(meta, store, "a", (0, 0, 0), backend=backend) - - -async def test_read_region_fancy_rejected(backend: str, store: Store) -> None: - _, meta = filled(store) - with pytest.raises(TypeError, match="only integers, slices"): - await read_region(meta, store, "a", ([0, 1], slice(None)), backend=backend) # type: ignore[arg-type] - - -async def test_read_region_out_of_bounds(backend: str, store: Store) -> None: - _, meta = filled(store) - with pytest.raises(IndexError, match="out of bounds"): - await read_region(meta, store, "a", (8, slice(None)), backend=backend) diff --git a/tests/crud/test_reference_backend.py b/tests/crud/test_reference_backend.py deleted file mode 100644 index 4fef43427b..0000000000 --- a/tests/crud/test_reference_backend.py +++ /dev/null @@ -1,73 +0,0 @@ -from __future__ import annotations - -from typing import Any - -import numpy as np -import pytest - -import zarr -from zarr.crud import NodeExistsError, get_backend -from zarr.errors import NodeNotFoundError -from zarr.storage import MemoryStore - - -def _array_meta() -> dict[str, Any]: - arr = zarr.create_array(store=MemoryStore(), shape=(8, 8), chunks=(4, 4), dtype="uint16") - return dict(arr.metadata.to_dict()) - - -async def test_reference_round_trip_chunk() -> None: - be = get_backend("reference") - store = MemoryStore() - meta = _array_meta() - await be.create_array(store, "a", meta, overwrite=False) - value = np.arange(16, dtype="uint16").reshape(4, 4) - await be.write_chunk(store, "a", meta, (0, 1), value.tobytes()) - raw = await be.read_chunk(store, "a", meta, (0, 1)) - np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(4, 4), value) - - -async def test_reference_read_subset_spans_chunks() -> None: - be = get_backend("reference") - store = MemoryStore() - arr = zarr.create_array(store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16") - data = np.arange(64, dtype="uint16").reshape(8, 8) - arr[:, :] = data - meta = dict(arr.metadata.to_dict()) - raw = await be.read_subset(store, "a", meta, (2, 1), (5, 4)) - np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(5, 4), data[2:7, 1:5]) - - -async def test_reference_create_exists_raises() -> None: - be = get_backend("reference") - store = MemoryStore() - meta = _array_meta() - await be.create_array(store, "a", meta, overwrite=False) - with pytest.raises(NodeExistsError): - await be.create_array(store, "a", meta, overwrite=False) - - -async def test_reference_read_metadata_missing_raises() -> None: - be = get_backend("reference") - with pytest.raises(NodeNotFoundError): - await be.read_metadata(MemoryStore(), "nope") - - -async def test_reference_v2_fortran_order_round_trip() -> None: - be = get_backend("reference") - store = MemoryStore() - arr = zarr.create_array( - store=store, name="f", shape=(4, 6), chunks=(4, 6), dtype="uint16", order="F", zarr_format=2 - ) - data = np.arange(24, dtype="uint16").reshape(4, 6) - arr[:, :] = data - meta = dict(arr.metadata.to_dict()) - meta.pop("attributes", None) - # read_chunk must return native C-contiguous bytes matching the logical data - raw = await be.read_chunk(store, "f", meta, (0, 0)) - np.testing.assert_array_equal(np.frombuffer(raw, dtype="uint16").reshape(4, 6), data) - # write_chunk must store data zarr-python reads back correctly - new = (data + 100).astype("uint16") - await be.write_chunk(store, "f", meta, (0, 0), np.ascontiguousarray(new).tobytes()) - back = zarr.open_array(store=store, path="f", mode="r") - np.testing.assert_array_equal(back[:, :], new) diff --git a/tests/crud/test_registry.py b/tests/crud/test_registry.py deleted file mode 100644 index f5a8f8b829..0000000000 --- a/tests/crud/test_registry.py +++ /dev/null @@ -1,39 +0,0 @@ -from __future__ import annotations - -import pytest - -from zarr.crud import CrudBackend, NodeExistsError, get_backend, register_backend - - -def test_node_exists_error_is_value_error() -> None: - assert issubclass(NodeExistsError, ValueError) - - -def test_default_backend_is_reference() -> None: - # the reference backend is registered at import and is the configured default - be = get_backend() - assert be is get_backend("reference") - - -def test_get_unknown_backend_raises() -> None: - with pytest.raises(KeyError, match="no CRUD backend"): - get_backend("does-not-exist") - - -def test_register_and_resolve_instance() -> None: - class Dummy: - pass - - dummy = Dummy() - register_backend("dummy-test", dummy) # type: ignore[arg-type] - try: - assert get_backend("dummy-test") is dummy # type: ignore[comparison-overlap] - finally: - from zarr.crud import _registry - - _registry._BACKENDS.pop("dummy-test", None) - - -def test_protocol_is_runtime_checkable() -> None: - # ReferenceBackend (registered as "reference") structurally satisfies the protocol - assert isinstance(get_backend("reference"), CrudBackend) diff --git a/tests/zarrs/__init__.py b/tests/zarrs/__init__.py deleted file mode 100644 index e69de29bb2..0000000000 diff --git a/tests/zarrs/conftest.py b/tests/zarrs/conftest.py deleted file mode 100644 index 092bce5473..0000000000 --- a/tests/zarrs/conftest.py +++ /dev/null @@ -1,46 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING, Any - -import pytest - -import zarr -from zarr.storage import LocalStore, MemoryStore - -if TYPE_CHECKING: - from collections.abc import AsyncIterator - from pathlib import Path - - from zarr.abc.store import Store - - -@pytest.fixture(params=["memory", "local"]) -async def store(request: pytest.FixtureRequest, tmp_path: Path) -> AsyncIterator[Store]: - """A writable store: MemoryStore exercises the generic Python-callback bridge, - LocalStore exercises the native zarrs filesystem store.""" - s: Store - if request.param == "memory": - s = await MemoryStore.open() - else: - s = await LocalStore.open(root=tmp_path / "store") - try: - yield s - finally: - s.close() - - -def array_metadata(**kwargs: Any) -> dict[str, Any]: - """Build an array metadata document using zarr-python itself, so the - documents fed to zarrs always match what zarr-python would write.""" - params: dict[str, Any] = { - "shape": (8, 8), - "chunks": (4, 4), - "dtype": "uint16", - "zarr_format": 3, - } | kwargs - arr = zarr.create_array(store=MemoryStore(), **params) - doc = dict(arr.metadata.to_dict()) - if params["zarr_format"] == 2: - # v2 attributes live in .zattrs, not in the .zarray document - doc.pop("attributes", None) - return doc diff --git a/tests/zarrs/test_bridge.py b/tests/zarrs/test_bridge.py deleted file mode 100644 index f997b052f2..0000000000 --- a/tests/zarrs/test_bridge.py +++ /dev/null @@ -1,53 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING - -import pytest - -pytest.importorskip( - "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError -) - -from zarr.storage import LocalStore, MemoryStore -from zarr.zarrs._bridge import StoreShim, resolve_store - -if TYPE_CHECKING: - from pathlib import Path - - -def test_shim_get_set_delete() -> None: - shim = StoreShim(MemoryStore()) - assert shim.get("a/b") is None - shim.set("a/b", b"xyz") - assert shim.get("a/b") == b"xyz" - assert shim.get_range("a/b", 1, 1) == b"y" - assert shim.get_range("a/b", 1, None) == b"yz" - assert shim.get_suffix("a/b", 2) == b"yz" - assert shim.getsize("a/b") == 3 - assert shim.getsize("missing") is None - assert shim.get_range("missing", 0, 1) is None - assert shim.get_suffix("missing", 1) is None - shim.delete("a/b") - assert shim.get("a/b") is None - - -def test_shim_listing() -> None: - shim = StoreShim(MemoryStore()) - shim.set("zarr.json", b"{}") - shim.set("a/zarr.json", b"{}") - shim.set("a/c/0/0", b"\x00") - assert shim.list() == ["a/c/0/0", "a/zarr.json", "zarr.json"] - assert shim.list_prefix("a/") == ["a/c/0/0", "a/zarr.json"] - assert shim.list_dir("a/") == (["a/zarr.json"], ["a/c/"]) - assert shim.list_dir("") == (["zarr.json"], ["a/"]) - assert shim.getsize_prefix("a/") == 3 - shim.delete_prefix("a/") - assert shim.list() == ["zarr.json"] - - -def test_resolve_store(tmp_path: Path) -> None: - local = LocalStore(tmp_path) - assert resolve_store(local) == {"filesystem": str(tmp_path)} - # read-only LocalStore must go through the shim so writes are rejected in Python - assert isinstance(resolve_store(LocalStore(tmp_path, read_only=True)), StoreShim) - assert isinstance(resolve_store(MemoryStore()), StoreShim) diff --git a/tests/zarrs/test_cache.py b/tests/zarrs/test_cache.py deleted file mode 100644 index 9af72555d8..0000000000 --- a/tests/zarrs/test_cache.py +++ /dev/null @@ -1,87 +0,0 @@ -from __future__ import annotations - -from typing import TYPE_CHECKING, Any - -import numpy as np -import pytest - -pytest.importorskip( - "_zarrs_bindings", reason="zarrs-bindings is not installed", exc_type=ImportError -) - -import _zarrs_bindings as zb - -import zarr -import zarr.zarrs # registers the "zarrs" CrudBackend -from zarr.crud import read_chunk, write_chunk -from zarr.storage import LocalStore, MemoryStore - -if TYPE_CHECKING: - from pathlib import Path - - -def _meta(store: Any, name: str = "a") -> dict[str, Any]: - arr = zarr.create_array(store=store, name=name, shape=(8, 8), chunks=(4, 4), dtype="uint16") - arr[:, :] = np.arange(64, dtype="uint16").reshape(8, 8) - return dict(arr.metadata.to_dict()) - - -@pytest.fixture(autouse=True) -def _clear_cache() -> None: - zb.clear_array_cache() - - -async def test_localstore_populates_cache(tmp_path: Path) -> None: - store = await LocalStore.open(root=tmp_path / "s") - meta = _meta(store) - assert zb.array_cache_len() == 0 - await read_chunk(meta, store, "a", (0, 0), backend="zarrs") - assert zb.array_cache_len() == 1 - # second op on the SAME array reuses the entry, does not grow the cache - await read_chunk(meta, store, "a", (1, 1), backend="zarrs") - assert zb.array_cache_len() == 1 - - -async def test_memorystore_is_not_cached() -> None: - store = MemoryStore() - meta = _meta(store) - await read_chunk(meta, store, "a", (0, 0), backend="zarrs") - assert zb.array_cache_len() == 0 - - -async def test_distinct_metadata_distinct_entries(tmp_path: Path) -> None: - store = await LocalStore.open(root=tmp_path / "s") - meta_a = _meta(store, "a") - meta_b = _meta(store, "b") - await read_chunk(meta_a, store, "a", (0, 0), backend="zarrs") - await read_chunk(meta_b, store, "b", (0, 0), backend="zarrs") - assert zb.array_cache_len() == 2 - - -async def test_cache_keyed_on_root_not_just_metadata(tmp_path: Path) -> None: - # two stores at different roots, identical metadata + path, different data. - # A correct cache (keyed on root) must return each store's own data. - s1 = await LocalStore.open(root=tmp_path / "s1") - s2 = await LocalStore.open(root=tmp_path / "s2") - a1 = zarr.create_array(store=s1, name="a", shape=(4, 4), chunks=(4, 4), dtype="uint16") - a1[:, :] = 1 - a2 = zarr.create_array(store=s2, name="a", shape=(4, 4), chunks=(4, 4), dtype="uint16") - a2[:, :] = 2 - meta = dict(a1.metadata.to_dict()) # identical metadata document - out1 = await read_chunk(meta, s1, "a", (0, 0), backend="zarrs") - out2 = await read_chunk(meta, s2, "a", (0, 0), backend="zarrs") - np.testing.assert_array_equal(out1, np.full((4, 4), 1, dtype="uint16")) - np.testing.assert_array_equal(out2, np.full((4, 4), 2, dtype="uint16")) - assert zb.array_cache_len() == 2 - - -async def test_cache_reflects_writes_through_store(tmp_path: Path) -> None: - # after the Array is cached, a write via the cached Array must be visible to - # a subsequent read (proves the cache does not stale-cache chunk data) - store = await LocalStore.open(root=tmp_path / "s") - meta = _meta(store) - await read_chunk(meta, store, "a", (0, 0), backend="zarrs") # caches the Array - new = np.full((4, 4), 99, dtype="uint16") - await write_chunk(meta, store, "a", (0, 0), new, backend="zarrs") # write via (cached) Array - out = await read_chunk(meta, store, "a", (0, 0), backend="zarrs") - np.testing.assert_array_equal(out, new) diff --git a/uv.lock b/uv.lock index ccc1f008b9..22c5c31f2c 100644 --- a/uv.lock +++ b/uv.lock @@ -4079,21 +4079,6 @@ zarrista = [ { name = "uv" }, { name = "zarrista" }, ] -zarrs = [ - { name = "coverage" }, - { name = "hypothesis" }, - { name = "numpydoc" }, - { name = "pytest" }, - { name = "pytest-accept" }, - { name = "pytest-asyncio" }, - { name = "pytest-benchmark" }, - { name = "pytest-codspeed" }, - { name = "pytest-cov" }, - { name = "pytest-xdist" }, - { name = "tomlkit" }, - { name = "uv" }, - { name = "zarrs-bindings" }, -] [package.metadata] requires-dist = [ @@ -4216,21 +4201,6 @@ zarrista = [ { name = "uv" }, { name = "zarrista", git = "https://github.com/developmentseed/zarrista?rev=95e47ad4c414c5920f0cf15550f923039641da8e" }, ] -zarrs = [ - { name = "coverage", specifier = ">=7.10" }, - { name = "hypothesis" }, - { name = "numpydoc" }, - { name = "pytest" }, - { name = "pytest-accept" }, - { name = "pytest-asyncio" }, - { name = "pytest-benchmark" }, - { name = "pytest-codspeed" }, - { name = "pytest-cov" }, - { name = "pytest-xdist" }, - { name = "tomlkit" }, - { name = "uv" }, - { name = "zarrs-bindings", directory = "packages/zarrs-bindings" }, -] [[package]] name = "zarr-metadata" @@ -4251,7 +4221,3 @@ source = { git = "https://github.com/developmentseed/zarrista?rev=95e47ad4c414c5 dependencies = [ { name = "zarr-metadata" }, ] - -[[package]] -name = "zarrs-bindings" -source = { directory = "packages/zarrs-bindings" } From 753d2dd537999078f49418c859ab7c156313780c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 21:29:43 +0200 Subject: [PATCH 60/75] fix(engine): correct basic-set widening and empty block-slice box A basic write whose dropped integer axis is not the leading axis (e.g. arr[:, 0] = v, reached via a numpy integer scalar) took the full-box fast path and broadcast the dimension-dropped value into the ndim-preserving box, failing when the axis was non-trailing. Re-insert a size-1 axis at each dropped integer axis before broadcasting. An empty block slice (blocks[1:0]) produced a SliceDimIndexer with start > stop, which _block_region mapped to a negative-length box; clamp end_exclusive to max(start, stop). Both were regressions from the engine rewiring, verified passing at b74b0c500. Adds deterministic regression tests. Assisted-by: ClaudeCode:claude-opus-4.8 --- src/zarr/core/array.py | 15 +++++++++--- tests/engine/test_asyncarray_wiring.py | 34 +++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 4 deletions(-) diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 33868d34cc..3bcc190950 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -5790,9 +5790,16 @@ def _prepare_set_selection( identity_box = None if identity_post: - # full-box write: broadcast the value into the box without a read + # full-box write: broadcast the value into the box without a read. Basic + # integer axes drop a dimension from the value (their post entry is a + # plain int, not an orthogonal `_Squeeze` marker), so re-insert a size-1 + # axis at each dropped position before broadcasting -- otherwise a + # non-trailing dropped axis (e.g. `arr[:, 0] = v`) leaves the value's + # shape misaligned with the ndim-preserving box. + int_axes = tuple(i for i, p in enumerate(stripped) if isinstance(p, (int, np.integer))) + broadcast_value = _widen_value_for_squeeze(assign_value, int_axes, len(region.shape)) identity_box = np.array( - np.broadcast_to(np.asarray(assign_value), region.shape), dtype=dtype + np.broadcast_to(np.asarray(broadcast_value), region.shape), dtype=dtype ) return _SetSelectionPrep( identity_box=identity_box, @@ -6018,7 +6025,9 @@ def _block_region( """ indexer = BlockIndexer(selection, shape, chunk_grid) starts = tuple(di.start for di in indexer.dim_indexers) - ends = tuple(di.stop for di in indexer.dim_indexers) + # an empty block slice (e.g. `blocks[1:0]`) yields `start > stop` on that + # axis; clamp so the box is zero-length rather than negative-length. + ends = tuple(max(di.start, di.stop) for di in indexer.dim_indexers) return Region(start=starts, end_exclusive=ends) diff --git a/tests/engine/test_asyncarray_wiring.py b/tests/engine/test_asyncarray_wiring.py index 90fc87c2ed..5725f57065 100644 --- a/tests/engine/test_asyncarray_wiring.py +++ b/tests/engine/test_asyncarray_wiring.py @@ -1,6 +1,6 @@ from __future__ import annotations -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, cast import numpy as np @@ -11,6 +11,7 @@ if TYPE_CHECKING: from zarr.core.buffer import BufferPrototype, NDArrayLike, NDBuffer + from zarr.core.indexing import BasicSelection from zarr.core.metadata import ArrayMetadata @@ -75,3 +76,34 @@ def test_strided_read_preserves_fortran_order() -> None: assert full.flags.f_contiguous assert strided.flags.f_contiguous np.testing.assert_array_equal(strided, np.arange(64.0).reshape(8, 8)[::2, ::2]) + + +def test_basic_set_integer_axis_widens_value() -> None: + # Regression: a basic write whose dropped integer axis is *not* the leading + # axis (e.g. `arr[:, 0] = v`) took the full-box fast path, which broadcast the + # dimension-dropped value straight into the ndim-preserving box -- (3,) could + # not broadcast to box shape (3, 1). A numpy integer scalar keeps the routing + # in the basic (not orthogonal) facade, matching the property-test example. + expected = np.zeros((3, 3), dtype="int64") + z = zarr.create_array(MemoryStore(), shape=(3, 3), chunks=(3, 3), dtype="int64") + z[:, :] = expected + value = np.array([1, 2, 3], dtype="int64") + # a numpy integer scalar (not a Python int) keeps `__setitem__` routing on the + # basic facade rather than the orthogonal one, reproducing the failing example. + selection = cast("BasicSelection", (slice(None), np.int64(0))) + z.set_basic_selection(selection, value) + expected[:, 0] = value + np.testing.assert_array_equal(np.asarray(z[:, :]), expected) + + +def test_empty_block_slice_reads_zero_length_box() -> None: + # Regression: an empty block slice (`blocks[1:0]`) produced a + # SliceDimIndexer with start > stop, which `_block_region` mapped to a + # negative-length box (start=1, end_exclusive=0) and crashed with + # "negative dimensions are not allowed". + data = np.arange(2, dtype="int64") + z = zarr.create_array(MemoryStore(), shape=(2,), chunks=(1,), dtype="int64") + z[:] = data + result = np.asarray(z.get_block_selection((slice(1, 0),))) + np.testing.assert_array_equal(result, data[2:2]) + assert result.shape == (0,) From 671af2d7a6fd70588b138d1168b926d1ef0a24eb Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 22:00:15 +0200 Subject: [PATCH 61/75] test/ci: engine differential suite and zarrista CI job Add tests/engine/test_differential.py, exercising reads/writes/fancy indexing/sharded reads through both the default and zarrista engines against a numpy reference, and .github/workflows/engine.yml to run tests/engine and tests/zarrista in CI with the zarrista uv group and a Rust toolchain. Adds the changelog fragment for the array-engine-protocol feature, including two behavior notes surfaced during the branch's review: structured-dtype single-field writes now preserve other fields, and strided/fancy selections are served via their bounding box (which can raise on a missing chunk in the box even when read_missing_chunks=False and no selected element falls in that chunk). Assisted-by: ClaudeCode:claude-sonnet-5 --- .github/workflows/engine.yml | 49 +++++++++++++ changes/4181.feature.md | 19 +++++ tests/engine/test_differential.py | 116 ++++++++++++++++++++++++++++++ 3 files changed, 184 insertions(+) create mode 100644 .github/workflows/engine.yml create mode 100644 changes/4181.feature.md create mode 100644 tests/engine/test_differential.py diff --git a/.github/workflows/engine.yml b/.github/workflows/engine.yml new file mode 100644 index 0000000000..d02544fac8 --- /dev/null +++ b/.github/workflows/engine.yml @@ -0,0 +1,49 @@ +name: Array engine + +on: + push: + branches: [ main ] + paths: + - 'src/zarr/**' + - 'tests/engine/**' + - 'tests/zarrista/**' + - 'pyproject.toml' + - '.github/workflows/engine.yml' + pull_request: + branches: [ main ] + paths: + - 'src/zarr/**' + - 'tests/engine/**' + - 'tests/zarrista/**' + - 'pyproject.toml' + - '.github/workflows/engine.yml' + workflow_dispatch: + +permissions: {} + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +jobs: + test: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + fetch-depth: 0 # hatch-vcs needs tags to compute zarr's version + persist-credentials: false + - name: Install Rust + uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable + - name: Install uv + uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + with: + python-version: '3.12' + - name: Sync zarrista group + run: uv sync --group zarrista + - name: Run engine tests + # the ubuntu runner image ships a Rust toolchain; the maturin build + # backend for zarrista is fetched by uv on demand + run: uv run --group zarrista pytest tests/engine tests/zarrista -v diff --git a/changes/4181.feature.md b/changes/4181.feature.md new file mode 100644 index 0000000000..c28fbf48ff --- /dev/null +++ b/changes/4181.feature.md @@ -0,0 +1,19 @@ +`Array` and `AsyncArray` now route data I/O through pluggable *array engines* +(`zarr.abc.engine.ArrayEngine` / `AsyncArrayEngine`). The default engine +preserves existing behavior on every store and format. Pass +`engine="zarrista"` (requires the `zarrista` package) to serve Zarr v3 arrays +on local-filesystem, obstore, or icechunk storage through the Rust `zarrs` +implementation. The experimental `zarr.crud` and `zarr.zarrs` modules and the +bundled `zarrs-bindings` crate are removed in favor of this interface. + +As part of this rewiring, writing a single field of a structured dtype (e.g. +`z["x"] = ...`) now preserves the values of the array's other fields; it +previously clobbered them. + +Behavior note: strided and fancy (orthogonal/coordinate) selections are now +served internally via the contiguous bounding box that covers the selected +indices, with the requested elements picked out of that box afterwards. For +sparse selections this can transfer more data from storage than before, and +with the `array.read_missing_chunks` config set to `False`, a missing chunk +that lies within the bounding box will raise even if no selected element +falls inside that chunk. diff --git a/tests/engine/test_differential.py b/tests/engine/test_differential.py new file mode 100644 index 0000000000..5b08110b54 --- /dev/null +++ b/tests/engine/test_differential.py @@ -0,0 +1,116 @@ +"""The same operations through both engines must agree with numpy and each other.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +import numpy as np +import pytest + +import zarr +from zarr.storage import LocalStore + +try: + import zarrista # noqa: F401 + + ENGINES = ["default", "zarrista"] +except ImportError: + ENGINES = ["default"] + +if TYPE_CHECKING: + from pathlib import Path + + import numpy.typing as npt + + from zarr.core.engine import EngineName + +SHAPE = (10, 9) +CHUNKS = (3, 4) + + +@pytest.fixture +def reference(tmp_path: Path) -> tuple[Path, npt.NDArray[np.float64]]: + z = zarr.create_array(LocalStore(tmp_path), shape=SHAPE, chunks=CHUNKS, dtype="float64") + data = np.arange(90, dtype="float64").reshape(SHAPE) + z[:, :] = data + return tmp_path, data + + +READS: list[tuple[int | slice, int | slice]] = [ + (slice(None), slice(None)), + (slice(2, 7), slice(1, 5)), + # negative-step slices are rejected by `Array.__getitem__` (see + # `NegativeStepError`); only positive step > 1 is supported, so this + # covers strided reads on both axes without hitting that restriction. + (slice(1, 9, 2), slice(None, None, 3)), + (3, slice(None)), + (-1, -2), + (slice(4, 4), slice(None)), +] + + +@pytest.mark.parametrize("engine", ENGINES) +@pytest.mark.parametrize("sel", READS) +def test_reads_match_numpy( + reference: tuple[Path, npt.NDArray[np.float64]], + engine: EngineName, + sel: tuple[int | slice, int | slice], +) -> None: + tmp_path, data = reference + z = zarr.open_array(LocalStore(tmp_path), engine=engine) + np.testing.assert_array_equal(np.asarray(z[sel]), data[sel]) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_fancy_reads_match_numpy( + reference: tuple[Path, npt.NDArray[np.float64]], engine: EngineName +) -> None: + tmp_path, data = reference + z = zarr.open_array(LocalStore(tmp_path), engine=engine) + np.testing.assert_array_equal( + np.asarray(z.oindex[np.array([7, 1, 4]), np.array([0, 8])]), + data[np.ix_([7, 1, 4], [0, 8])], + ) + np.testing.assert_array_equal( + np.asarray(z.vindex[np.array([9, 0, 3]), np.array([8, 0, 2])]), + data[np.array([9, 0, 3]), np.array([8, 0, 2])], + ) + np.testing.assert_array_equal(np.asarray(z.blocks[1, 2]), data[3:6, 8:9]) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_writes_match_numpy( + reference: tuple[Path, npt.NDArray[np.float64]], engine: EngineName +) -> None: + tmp_path, data = reference + z = zarr.open_array(LocalStore(tmp_path), engine=engine) + expected = data.copy() + + z[0:3, 0:4] = 7.0 # aligned full chunk + expected[0:3, 0:4] = 7.0 + z[4:6, 2:9] = np.arange(14, dtype="float64").reshape(2, 7) # partial chunks + expected[4:6, 2:9] = np.arange(14, dtype="float64").reshape(2, 7) + z[1:9:3, ::4] = -1.0 # strided write (facade RMW) + expected[1:9:3, ::4] = -1.0 + + np.testing.assert_array_equal(np.asarray(z[:, :]), expected) + + +@pytest.mark.parametrize("engine", ENGINES) +def test_sharded_reads( + reference: tuple[Path, npt.NDArray[np.float64]], + engine: EngineName, + tmp_path_factory: pytest.TempPathFactory, +) -> None: + path = tmp_path_factory.mktemp("sharded") + z = zarr.create_array( + LocalStore(path), + shape=SHAPE, + chunks=(3, 4), # inner chunks + shards=(6, 8), # shard shape + dtype="int32", + ) + data = np.arange(90, dtype="int32").reshape(SHAPE) + z[:, :] = data + zr = zarr.open_array(LocalStore(path), engine=engine) + np.testing.assert_array_equal(np.asarray(zr[2:8, 3:9]), data[2:8, 3:9]) From 48b19264a160118a216748f0c794aa1b442bb943 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 23:31:57 +0200 Subject: [PATCH 62/75] fix(engine): keep device buffers in their own array namespace The engine facade forced host-numpy conversions (`np.asarray`, `np.array`, `np.ascontiguousarray`) on the engine result, which cupy-style device buffers reject via `__array__`. Post-index, field selection, memory-order materialization, broadcast, patch, and the contiguous-box wrap now stay in the result's own namespace (`astype`/`.copy()`/plain indexing, which cupy/torch implement identically), and empty-selection short circuits allocate through `prototype.nd_buffer.empty(...)` so a GPU prototype is honoured. No protocol change; plain-numpy behaviour is unchanged. A CPU-runnable regression drives `__setitem__` (identity and strided) and strided `__getitem__` through a stub engine returning an `NDArrayLike` stand-in whose `__array__` raises `TypeError`; it fails before this change and passes after. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 48 +++++-- src/zarr/core/engine/_normalize.py | 20 ++- tests/engine/test_device_buffer_facade.py | 163 ++++++++++++++++++++++ 3 files changed, 212 insertions(+), 19 deletions(-) create mode 100644 tests/engine/test_device_buffer_facade.py diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 3bcc190950..acdc469e24 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -5632,13 +5632,20 @@ def _finish_get_selection( # unchanged so a custom `NDArrayLike` type (e.g. GPU/torch buffers) # survives instead of being coerced to numpy by `np.asarray` return _finalize_result(raw, out, scalarize=scalarize) - box = np.asarray(raw) + # Stay in the engine result's own array namespace: field selection and + # `apply_post_index` are plain indexing (which cupy/torch implement), so a + # device buffer is never forced onto the host by an `np.asarray` here. + box = raw if fields: # non-empty `fields` selects structured sub-fields; an empty list/tuple # (from `pop_fields` on a field-free selection) means "all fields" box = box[fields] # type: ignore[index] result = apply_post_index(box, post_index) - result = np.asarray(result, order=order) + # Materialize the post-indexed (strided/fancy) view in the array's effective + # memory order without leaving the result's namespace -- `astype` is a method + # on the array itself, so a device buffer stays on-device (unlike + # `np.asarray(..., order=...)`, which would coerce it to numpy). + result = result.astype(result.dtype, order=order, copy=True) return _finalize_result(result, out, scalarize=scalarize) @@ -5666,7 +5673,9 @@ async def _get_selection( metadata, config, region, post_index, out=out, fields=fields ) if product(region.shape) == 0: - empty = np.empty(result_shape, dtype=out_dtype) + # allocate through the prototype so an empty read honours the requested + # buffer type (e.g. a GPU prototype) instead of a host numpy array + empty = prototype.nd_buffer.empty(result_shape, out_dtype).as_ndarray_like() return _finalize_result(empty, out, scalarize=scalarize) raw = await engine.read_selection(region, prototype=prototype) @@ -5695,7 +5704,9 @@ def _get_selection_sync( metadata, config, region, post_index, out=out, fields=fields ) if product(region.shape) == 0: - empty = np.empty(result_shape, dtype=out_dtype) + # allocate through the prototype so an empty read honours the requested + # buffer type (e.g. a GPU prototype) instead of a host numpy array + empty = prototype.nd_buffer.empty(result_shape, out_dtype).as_ndarray_like() return _finalize_result(empty, out, scalarize=scalarize) raw = engine.read_selection(region, prototype=prototype) @@ -5798,9 +5809,11 @@ def _prepare_set_selection( # shape misaligned with the ndim-preserving box. int_axes = tuple(i for i, p in enumerate(stripped) if isinstance(p, (int, np.integer))) broadcast_value = _widen_value_for_squeeze(assign_value, int_axes, len(region.shape)) - identity_box = np.array( - np.broadcast_to(np.asarray(broadcast_value), region.shape), dtype=dtype - ) + # `np.broadcast_to` dispatches through `__array_function__`, so a device + # buffer stays in its own namespace; `astype` (a method on the array) + # then materializes a writable, dtype-correct box without an + # `np.asarray` host coercion. + identity_box = np.broadcast_to(broadcast_value, region.shape).astype(dtype, copy=True) return _SetSelectionPrep( identity_box=identity_box, fields=fields, @@ -5814,8 +5827,13 @@ def _patch_selection_box(raw: NDArrayLike, prep: _SetSelectionPrep) -> NDArrayLi (`prep.identity_box is None`), using the same `post_index` the read path uses (dropping the orthogonal `_Squeeze` marker and widening the value at dropped integer axes, matching `oindex_set`).""" - box = np.array(np.asarray(raw)) - target = box[prep.fields] if prep.fields else box # type: ignore[index] + # `.copy()` yields a writable box in the engine result's own namespace (a + # device buffer is patched on-device); `np.asarray` would coerce it to host. + box = raw.copy() + # `target` indexing/assignment uses tuple/ellipsis keys the `NDArrayLike` + # protocol does not type (it only declares slice-key access), so it is + # `Any`-typed here; at runtime numpy/cupy/torch all support these keys. + target: Any = box[prep.fields] if prep.fields else box # type: ignore[index] if prep.stripped == (): target[...] = prep.assign_value else: @@ -5825,10 +5843,14 @@ def _patch_selection_box(raw: NDArrayLike, prep: _SetSelectionPrep) -> NDArrayLi def _finish_set_selection(box: NDArrayLike, prototype: BufferPrototype) -> NDBuffer: """Shared postamble for `_set_selection`/`_set_selection_sync`: make the - box contiguous (0-d arrays pass through unchanged -- `np.ascontiguousarray` - would otherwise promote them to shape `(1,)`) and wrap it for - `engine.write_selection`.""" - contiguous_box = box if box.ndim == 0 else np.ascontiguousarray(box) + box C-contiguous and wrap it for `engine.write_selection`. + + `astype` (a method on the array) forces C-order in the box's own namespace + -- copying only when needed, exactly like `np.ascontiguousarray` -- so a + device buffer (cupy/torch) stays on-device instead of being coerced to host. + 0-d arrays pass through unchanged (`np.ascontiguousarray` would have promoted + them to shape `(1,)`).""" + contiguous_box = box if box.ndim == 0 else box.astype(box.dtype, order="C", copy=False) return prototype.nd_buffer.from_ndarray_like(contiguous_box) diff --git a/src/zarr/core/engine/_normalize.py b/src/zarr/core/engine/_normalize.py index cc743cd8a3..33edf47162 100644 --- a/src/zarr/core/engine/_normalize.py +++ b/src/zarr/core/engine/_normalize.py @@ -9,13 +9,14 @@ import operator import types -from typing import TYPE_CHECKING, Any +from typing import TYPE_CHECKING, Any, cast import numpy as np from zarr.abc.engine import Region if TYPE_CHECKING: + from zarr.core.buffer import NDArrayLike from zarr.core.indexing import BasicSelection @@ -152,14 +153,21 @@ def normalize_orthogonal(selection: Any, shape: tuple[int, ...]) -> tuple[Region return region, post -def apply_post_index(box: np.ndarray, post: tuple[Any, ...]) -> np.ndarray: - """Apply a post index produced by a `normalize_*` helper to a box read.""" +def apply_post_index(box: NDArrayLike, post: tuple[Any, ...]) -> NDArrayLike: + """Apply a post index produced by a `normalize_*` helper to a box read. + + Indexing stays in `box`'s own array namespace (numpy, cupy, torch, ...) so a + device buffer is never forced onto the host; the local is `Any`-typed because + the `NDArrayLike` protocol only types slice-key indexing, not the tuple keys + the `normalize_*` helpers emit. + """ + indexed: Any = box if post and isinstance(post[-1], _Squeeze): - result = box[post[:-1]] if len(post) > 1 else box - return np.squeeze(result, axis=post[-1].axes) + result = indexed[post[:-1]] if len(post) > 1 else indexed + return cast("NDArrayLike", np.squeeze(result, axis=post[-1].axes)) if post == (): return box - return np.asarray(box[post]) + return cast("NDArrayLike", indexed[post]) def strip_squeeze(post: tuple[Any, ...]) -> tuple[Any, ...]: diff --git a/tests/engine/test_device_buffer_facade.py b/tests/engine/test_device_buffer_facade.py new file mode 100644 index 0000000000..7d97495789 --- /dev/null +++ b/tests/engine/test_device_buffer_facade.py @@ -0,0 +1,163 @@ +"""Regression tests: the engine facade must not force a host (numpy) conversion +on a device buffer (e.g. cupy/torch) whose implicit `np.asarray` coercion is +refused. + +A `_DeviceArray` stand-in wraps a numpy array but raises `TypeError` from +`__array__`, exactly like cupy refusing an implicit host copy. Everything else +(indexing, `astype`, `copy`, and numpy's `__array_function__` protocol) is +delegated to the wrapped array and re-wrapped, so operations that *stay in the +array's own namespace* keep working while any `np.asarray`/`np.array` coercion +raises. Driving `Array.__setitem__`/`__getitem__` through a stub engine that +returns such buffers therefore fails before the facade's namespace fixes and +passes after them. +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any + +import numpy as np +import numpy.typing as npt +import pytest + +import zarr +from zarr.storage import MemoryStore + +if TYPE_CHECKING: + from zarr.abc.engine import Region + from zarr.core.buffer import BufferPrototype, NDBuffer + from zarr.core.metadata import ArrayMetadata + + +class _DeviceArray: + """A minimal `NDArrayLike` whose implicit host conversion is refused. + + Mimics a cupy array: indexing / `astype` / `copy` and numpy function + dispatch (`__array_function__`) all work in-namespace, but `__array__` + (what `np.asarray`/`np.array` call) raises, so any host coercion blows up. + """ + + def __init__(self, data: npt.NDArray[Any]) -> None: + self._a: npt.NDArray[Any] = data + + # --- host coercion is forbidden ------------------------------------- + def __array__(self, dtype: Any = None) -> npt.NDArray[Any]: + raise TypeError("implicit conversion to a host numpy array is not allowed") + + # --- in-namespace numpy-function dispatch --------------------------- + def __array_function__( + self, func: Any, types: Any, args: tuple[Any, ...], kwargs: dict[str, Any] + ) -> Any: + unwrapped = tuple(a._a if isinstance(a, _DeviceArray) else a for a in args) + result = func(*unwrapped, **kwargs) + if isinstance(result, np.ndarray): + return _DeviceArray(result) + return result + + # --- ndarray-like surface ------------------------------------------- + @property + def dtype(self) -> np.dtype[Any]: + return self._a.dtype + + @property + def shape(self) -> tuple[int, ...]: + return self._a.shape + + @property + def ndim(self) -> int: + return self._a.ndim + + @property + def size(self) -> int: + return self._a.size + + def __len__(self) -> int: + return len(self._a) + + def __getitem__(self, key: Any) -> _DeviceArray: + return _DeviceArray(self._a[key]) + + def __setitem__(self, key: Any, value: Any) -> None: + self._a[key] = value._a if isinstance(value, _DeviceArray) else value + + def astype(self, dtype: Any, order: Any = "K", *, copy: bool = True) -> _DeviceArray: + return _DeviceArray(self._a.astype(dtype, order=order, copy=copy)) + + def copy(self) -> _DeviceArray: + return _DeviceArray(self._a.copy()) + + def reshape(self, *args: Any, **kwargs: Any) -> _DeviceArray: + return _DeviceArray(self._a.reshape(*args, **kwargs)) + + +def _region_to_index(region: Region) -> tuple[slice, ...]: + return tuple(slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True)) + + +class _DeviceEngine: + """A synchronous `ArrayEngine` backed by numpy that hands the facade + `_DeviceArray` buffers (on read) and unwraps them (on write).""" + + def __init__(self, data: np.ndarray[Any, Any]) -> None: + self._data = data + + def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> _DeviceArray: + return _DeviceArray(self._data[_region_to_index(selection)].copy()) + + def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + arr: object = value.as_ndarray_like() + assert isinstance(arr, _DeviceArray), ( + "facade coerced the box off the device namespace before writing" + ) + self._data[_region_to_index(selection)] = arr._a + + def with_metadata(self, metadata: ArrayMetadata) -> _DeviceEngine: + return self + + +def _array_on_device_engine() -> tuple[zarr.Array[Any], _DeviceEngine]: + z = zarr.create_array(MemoryStore(), shape=(8,), chunks=(4,), dtype="int64") + engine = _DeviceEngine(np.zeros(8, dtype="int64")) + # inject the device engine as this array's cached sync engine + z._engine = engine # type: ignore[assignment] + return z, engine + + +def test_identity_setitem_keeps_device_buffer() -> None: + # full-box write: `_prepare_set_selection` broadcasts the value into the box + # without a read; it must not `np.asarray` the device value. + z, engine = _array_on_device_engine() + value = _DeviceArray(np.arange(8, dtype="int64")) + z[:] = value + np.testing.assert_array_equal(engine._data, np.arange(8, dtype="int64")) + + +def test_strided_setitem_reads_and_patches_on_device() -> None: + # strided write is a read-modify-write: the engine box read comes back as a + # `_DeviceArray`, and `_patch_selection_box` must patch it in-namespace + # (`raw.copy()`), never `np.asarray(raw)`. + z, engine = _array_on_device_engine() + z[::2] = np.array([10, 20, 30, 40], dtype="int64") + expected = np.zeros(8, dtype="int64") + expected[::2] = [10, 20, 30, 40] + np.testing.assert_array_equal(engine._data, expected) + + +def test_strided_getitem_keeps_device_buffer() -> None: + # strided read: `_finish_get_selection` applies the post index and reorders + # the result in the device namespace, returning a `_DeviceArray` untouched + # by `np.asarray`. + z, engine = _array_on_device_engine() + engine._data[:] = np.arange(8, dtype="int64") + result: object = z[::2] + assert isinstance(result, _DeviceArray), "strided read coerced off the device namespace" + np.testing.assert_array_equal(result._a, np.arange(8, dtype="int64")[::2]) + + +def test_facade_never_coerces_device_buffer_to_host() -> None: + # Belt-and-braces: a bare `np.asarray` on the stand-in must raise, proving the + # tests above would trip any host coercion the facade performed. + with pytest.raises(TypeError): + np.asarray(_DeviceArray(np.arange(4))) From dde203c4f91ad6d4d8f22f6e26e6ecb67c7f166c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 23:32:14 +0200 Subject: [PATCH 63/75] fix(engine): preserve the engine spec across with_config/update_attributes `AsyncArray.with_config`, `Array.with_config`, and `Array.update_attributes` rebuilt the array wrapper without threading the engine, so a custom or named engine was silently replaced by the default one. `AsyncArray` now keeps the ORIGINAL engine spec (name/instance/None) alongside its resolved engine, and all three paths re-thread that spec so the same engine family is re-resolved against the new config/metadata. The original spec (not the resolved engine) is passed, so a named engine is rebuilt with the new config rather than freezing the old one. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 26 +++++++- tests/engine/test_engine_spec_preservation.py | 64 +++++++++++++++++++ 2 files changed, 87 insertions(+), 3 deletions(-) create mode 100644 tests/engine/test_engine_spec_preservation.py diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index acdc469e24..3fa337e4dd 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -351,6 +351,13 @@ class AsyncArray[T_ArrayMetadata: (ArrayV2Metadata, ArrayV3Metadata)]: # derived, per-array data path; excluded from equality/repr like other # engine instances lack value semantics (two default engines are never `==`) engine: AsyncArrayEngine = field(init=False, compare=False, repr=False) + # the ORIGINAL engine spec (name/instance/None) this array was built with, + # kept so `with_config` can re-resolve the same engine family against the new + # config instead of freezing the already-resolved engine (which would carry + # the old config); excluded from equality/repr for the same reason as `engine` + _engine_spec: AsyncArrayEngine | EngineName | None = field( + init=False, compare=False, repr=False + ) _chunk_grid: ChunkGrid = field(init=False) config: ArrayConfig @@ -391,6 +398,7 @@ def __init__( "codec_pipeline", create_codec_pipeline(metadata=metadata_parsed, store=store_path.store), ) + object.__setattr__(self, "_engine_spec", engine) object.__setattr__( self, "engine", @@ -1218,7 +1226,15 @@ def with_config(self, config: ArrayConfigLike) -> Self: # Merge new config with existing config, so missing keys are inherited # from the current array rather than from global defaults new_config = ArrayConfig(**{**self.config.to_dict(), **config}) # type: ignore[arg-type] - return type(self)(metadata=self.metadata, store_path=self.store_path, config=new_config) + # re-resolve the ORIGINAL engine spec against the new config (a named + # engine gets rebuilt with `new_config`; an instance is returned as-is). + # Passing `self.engine` here would freeze the old config into the copy. + return type(self)( + metadata=self.metadata, + store_path=self.store_path, + config=new_config, + engine=self._engine_spec, + ) async def nchunks_initialized(self) -> int: """ @@ -2306,7 +2322,9 @@ def with_config(self, config: ArrayConfigLike) -> Self: ------- A new Array """ - return type(self)(self._async_array.with_config(config)) + # preserve this array's original engine spec so the new copy re-resolves + # the same engine family (the async array carries its own spec too) + return type(self)(self._async_array.with_config(config), engine_spec=self._engine_spec) @property def nbytes(self) -> int: @@ -4039,7 +4057,9 @@ def update_attributes(self, new_attributes: dict[str, JSON]) -> Self: overwritten by the new values. """ new_array = sync(self.async_array.update_attributes(new_attributes)) - return type(self)(new_array) + # carry the original engine spec so the new wrapper re-resolves the same + # engine family rather than silently falling back to the default engine + return type(self)(new_array, engine_spec=self._engine_spec) def __repr__(self) -> str: return f"" diff --git a/tests/engine/test_engine_spec_preservation.py b/tests/engine/test_engine_spec_preservation.py new file mode 100644 index 0000000000..0ee6fdd511 --- /dev/null +++ b/tests/engine/test_engine_spec_preservation.py @@ -0,0 +1,64 @@ +"""The original engine spec must survive `with_config` / `update_attributes`. + +`with_config` and `update_attributes` build a fresh array wrapper; each must +re-thread the array's engine spec so a custom engine is not silently dropped in +favour of the default one. Each path stores the ORIGINAL spec (name/instance), +so a named engine is re-resolved against the new config and an instance is +carried through unchanged -- these tests use instances and assert identity, +which is exactly what regresses when the spec is dropped (the copy would fall +back to a freshly-resolved default engine instead). +""" + +from __future__ import annotations + +from typing import Any + +import zarr +from zarr.core.array import AsyncArray +from zarr.core.engine import DefaultArrayEngine, DefaultAsyncArrayEngine +from zarr.storage import MemoryStore + + +def _async_array_with_custom_engine() -> tuple[AsyncArray[Any], DefaultAsyncArrayEngine]: + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + aa = z.async_array + custom = DefaultAsyncArrayEngine( + store_path=aa.store_path, metadata=aa.metadata, config=aa.config + ) + built = AsyncArray( + metadata=aa.metadata, store_path=aa.store_path, config=aa.config, engine=custom + ) + return built, custom + + +def test_asyncarray_with_config_preserves_engine() -> None: + built, custom = _async_array_with_custom_engine() + assert built.engine is custom + copy = built.with_config({"order": "F"}) + # the custom engine spec is re-resolved (an instance is returned unchanged), + # so the copy keeps it instead of falling back to a default engine + assert copy.engine is custom + + +def test_array_with_config_preserves_engine() -> None: + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + aa = z.async_array + custom = DefaultArrayEngine( + DefaultAsyncArrayEngine(store_path=aa.store_path, metadata=aa.metadata, config=aa.config) + ) + arr = zarr.Array(aa, engine_spec=custom) + assert arr.engine is custom + copy = arr.with_config({"order": "F"}) + assert copy.engine is custom + + +def test_array_update_attributes_preserves_engine() -> None: + z = zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") + aa = z.async_array + custom = DefaultArrayEngine( + DefaultAsyncArrayEngine(store_path=aa.store_path, metadata=aa.metadata, config=aa.config) + ) + arr = zarr.Array(aa, engine_spec=custom) + assert arr.engine is custom + updated = arr.update_attributes({"foo": "bar"}) + assert updated.engine is custom From b3011513f3eef115738647e70c9eb2e287545df2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 23:32:34 +0200 Subject: [PATCH 64/75] fix(engine): validate out= against the coordinate selection shape `get_coordinate_selection` flattens the coordinate arrays into a pointwise index, so the raw engine read is 1-d; the `out`-shape check therefore rejected a multi-dimensional `out` that the pre-engine implementation accepted (e.g. a 2-d coordinate array with a matching 2-d `out`). The pointwise result is now reshaped back to the selection shape and `out` is validated/filled against that shape via a shared `_finalize_coordinate_result` helper. The 1-d case is unchanged. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 49 +++++++++++++++++++------- tests/engine/test_coordinate_out.py | 54 +++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+), 13 deletions(-) create mode 100644 tests/engine/test_coordinate_out.py diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 3fa337e4dd..801a9daba0 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -1580,13 +1580,11 @@ async def get_coordinate_selection( if prototype is None: prototype = default_buffer_prototype() region, post, sel_shape = _coordinate_region_post(selection, self.metadata.shape) - out_array = await self._get_selection( - region, post, prototype=prototype, out=out, fields=fields - ) - if hasattr(out_array, "shape"): - # restore the (possibly multi-dimensional) selection shape - out_array = cast("NDArrayLikeOrScalar", np.asarray(out_array).reshape(sel_shape)) - return out_array + # `out` is validated/filled against the (possibly multi-dimensional) + # selection shape, not the flattened pointwise read, so it is handled by + # `_finalize_coordinate_result` rather than the flat-shaped inner read. + out_array = await self._get_selection(region, post, prototype=prototype, fields=fields) + return _finalize_coordinate_result(out_array, sel_shape, out) async def _save_metadata(self, metadata: ArrayMetadata, ensure_parents: bool = False) -> None: """ @@ -3584,21 +3582,19 @@ def get_coordinate_selection( if prototype is None: prototype = default_buffer_prototype() region, post, sel_shape = _coordinate_region_post(selection, self.shape) + # `out` is validated/filled against the (possibly multi-dimensional) + # selection shape, not the flattened pointwise read, so it is handled by + # `_finalize_coordinate_result` rather than the flat-shaped inner read. out_array = _get_selection_sync( self.engine, self.metadata, self.config, region, post, - out=out, fields=fields, prototype=prototype, ) - - if hasattr(out_array, "shape"): - # restore shape - out_array = np.asarray(out_array).reshape(sel_shape) - return out_array + return _finalize_coordinate_result(out_array, sel_shape, out) def set_coordinate_selection( self, @@ -5602,6 +5598,33 @@ def _finalize_result( return result +def _finalize_coordinate_result( + out_array: NDArrayLikeOrScalar, sel_shape: tuple[int, ...], out: NDBuffer | None +) -> NDArrayLikeOrScalar: + """Reshape a flat pointwise (coordinate/mask) read back to the selection + shape and, if an `out` buffer was supplied, validate and fill it. + + `_coordinate_region_post` flattens the (possibly multi-dimensional) + coordinate arrays before handing the engine a pointwise index, so the raw + read is 1-d. `out` is validated against the SELECTION shape -- which may be + multi-dimensional -- not that flattened shape, matching the pre-engine + behaviour where a 2-d coordinate array accepted a matching 2-d `out`. + """ + if hasattr(out_array, "shape"): + # restore the (possibly multi-dimensional) selection shape + out_array = cast("NDArrayLikeOrScalar", np.asarray(out_array).reshape(sel_shape)) + if out is not None: + if not isinstance(out, NDBuffer): + raise TypeError(f"out argument needs to be an NDBuffer. Got {type(out)!r}") + if out.shape != sel_shape: + raise ValueError( + f"shape of out argument doesn't match. Expected {sel_shape}, got {out.shape}" + ) + out.as_ndarray_like()[...] = out_array # type: ignore[index] + return out.as_ndarray_like() + return out_array + + def _get_selection_prepare( metadata: ArrayMetadata, config: ArrayConfig, diff --git a/tests/engine/test_coordinate_out.py b/tests/engine/test_coordinate_out.py new file mode 100644 index 0000000000..20b6a66605 --- /dev/null +++ b/tests/engine/test_coordinate_out.py @@ -0,0 +1,54 @@ +"""`get_coordinate_selection(..., out=...)` must validate `out` against the +selection shape, which may be multi-dimensional. + +The engine facade flattens the coordinate arrays into a pointwise index, so the +raw read is 1-d; a naive `out`-shape check against that flattened shape rejected +a multi-dimensional `out` that the pre-engine implementation accepted. These +tests pin both the multi-dimensional and the flat cases. +""" + +from __future__ import annotations + +from typing import Any + +import numpy as np +import pytest + +import zarr +from zarr.core.buffer import default_buffer_prototype +from zarr.storage import MemoryStore + + +def _filled_array() -> zarr.Array[Any]: + z = zarr.create_array(MemoryStore(), shape=(5, 5), chunks=(5, 5), dtype="int32") + z[:] = np.arange(25, dtype="int32").reshape(5, 5) + return z + + +def test_coordinate_selection_out_multidim() -> None: + z = _filled_array() + coords = (np.array([[0, 1], [2, 3]]), np.array([[0, 1], [2, 3]])) + out = default_buffer_prototype().nd_buffer.from_numpy_array(np.zeros((2, 2), dtype="int32")) + result = z.get_coordinate_selection(coords, out=out) + expected = np.array([[0, 6], [12, 18]], dtype="int32") + np.testing.assert_array_equal(np.asarray(result), expected) + np.testing.assert_array_equal(out.as_numpy_array(), expected) + + +def test_coordinate_selection_out_1d() -> None: + z = _filled_array() + coords = (np.array([0, 1, 4]), np.array([0, 1, 4])) + out = default_buffer_prototype().nd_buffer.from_numpy_array(np.zeros((3,), dtype="int32")) + result = z.get_coordinate_selection(coords, out=out) + expected = np.array([0, 6, 24], dtype="int32") + np.testing.assert_array_equal(np.asarray(result), expected) + np.testing.assert_array_equal(out.as_numpy_array(), expected) + + +def test_coordinate_selection_out_shape_mismatch_raises() -> None: + z = _filled_array() + coords = (np.array([[0, 1], [2, 3]]), np.array([[0, 1], [2, 3]])) + # a flat out no longer matches the 2-d selection shape + out = default_buffer_prototype().nd_buffer.from_numpy_array(np.zeros((4,), dtype="int32")) + with pytest.raises(ValueError, match="shape of out argument"): + z.get_coordinate_selection(coords, out=out) From 92f82e3a20dbde311f5fce189969a1d9ca55019d Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 23:32:50 +0200 Subject: [PATCH 65/75] refactor(engine): drop the unused normalize_block helper `normalize_block` was only ever exercised by its own test; block selections are served through `_block_region` (which reuses `BlockIndexer` for slice/irregular grids that `normalize_block` could not handle). Remove the helper, its exports, and its test. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/engine/__init__.py | 2 -- src/zarr/core/engine/_normalize.py | 19 ------------------- tests/engine/test_normalize.py | 8 -------- 3 files changed, 29 deletions(-) diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py index 0d99cf4f0b..23b626aa29 100644 --- a/src/zarr/core/engine/__init__.py +++ b/src/zarr/core/engine/__init__.py @@ -7,7 +7,6 @@ from zarr.core.engine._normalize import ( apply_post_index, normalize_basic, - normalize_block, normalize_coordinate, normalize_orthogonal, squeeze_axes, @@ -30,7 +29,6 @@ "apply_post_index", "classify_engine_arg", "normalize_basic", - "normalize_block", "normalize_coordinate", "normalize_orthogonal", "resolve_async_engine", diff --git a/src/zarr/core/engine/_normalize.py b/src/zarr/core/engine/_normalize.py index 33edf47162..a6c11d5ea0 100644 --- a/src/zarr/core/engine/_normalize.py +++ b/src/zarr/core/engine/_normalize.py @@ -211,22 +211,3 @@ def normalize_coordinate( ends = tuple(int(c.max()) + 1 if c.size else 0 for c in coords) post = tuple(c - s for c, s in zip(coords, starts, strict=True)) return Region(start=starts, end_exclusive=ends), post - - -def normalize_block( - block_coords: tuple[int, ...], - *, - chunk_grid_shape: tuple[int, ...], - chunk_shape: tuple[int, ...], - shape: tuple[int, ...], -) -> Region: - """Normalize a block (chunk-grid) selection to its contiguous box.""" - starts = [] - ends = [] - for dim, (b, nblocks, csize, size) in enumerate( - zip(block_coords, chunk_grid_shape, chunk_shape, shape, strict=True) - ): - idx = _normalize_int(b, nblocks, dim) - starts.append(idx * csize) - ends.append(min((idx + 1) * csize, size)) - return Region(start=tuple(starts), end_exclusive=tuple(ends)) diff --git a/tests/engine/test_normalize.py b/tests/engine/test_normalize.py index 90411f2b42..587a4c2a05 100644 --- a/tests/engine/test_normalize.py +++ b/tests/engine/test_normalize.py @@ -9,7 +9,6 @@ from zarr.core.engine import ( apply_post_index, normalize_basic, - normalize_block, normalize_coordinate, normalize_orthogonal, strip_squeeze, @@ -76,13 +75,6 @@ def test_normalize_coordinate_matches_numpy_vindex() -> None: np.testing.assert_array_equal(apply_post_index(_read_box(region), post), ARR[coords]) -def test_normalize_block_is_contiguous() -> None: - # chunk shape (3, 4) over SHAPE (10, 9): block (1, 2) spans rows 3:6, cols 8:9 - region = normalize_block((1, 2), chunk_grid_shape=(4, 3), chunk_shape=(3, 4), shape=SHAPE) - assert region.start == (3, 8) - assert region.end_exclusive == (6, 9) - - def test_normalize_basic_rejects_fancy() -> None: with pytest.raises(TypeError): normalize_basic((np.array([1, 2]), slice(None)), SHAPE) # type: ignore[arg-type] From be7fd09b323ac48d02e65b06a3abeaea21461272 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 23:33:11 +0200 Subject: [PATCH 66/75] docs(engine): API reference pages, changelog, and spec cleanups - Add mkdocstrings pages for `zarr.abc.engine` and `zarr.zarrista`, wired into the mkdocs nav. - Changelog: note that partial (non-full-box) writes now do an internal read-modify-write, so `read_missing_chunks: False` raises `ChunkNotFoundError` on uninitialized chunks; drop the stale claim that `zarr.crud`/`zarr.zarrs`/ `zarrs-bindings` were removed (they never shipped to users). - Spec doc: add the `config` parameter to the hierarchy-protocol snippet and reword the stale "registered on import" note to describe lazy resolver import. - Drop the obsolete `packages/zarrs-bindings/target/` .gitignore entry. Assisted-by: ClaudeCode:claude-opus-4-8 --- .gitignore | 3 --- changes/4181.feature.md | 8 +++++--- docs/api/zarr/abc/engine.md | 5 +++++ docs/api/zarr/zarrista.md | 5 +++++ .../specs/2026-07-22-array-engine-protocol-design.md | 7 ++++--- mkdocs.yml | 2 ++ 6 files changed, 21 insertions(+), 9 deletions(-) create mode 100644 docs/api/zarr/abc/engine.md create mode 100644 docs/api/zarr/zarrista.md diff --git a/.gitignore b/.gitignore index ae184fa731..3284865d6c 100644 --- a/.gitignore +++ b/.gitignore @@ -94,6 +94,3 @@ zarr.egg-info/ # zarr-metadata package lockfile (a library, not an app) packages/zarr-metadata/uv.lock - -# zarrs-bindings Rust build artifacts -packages/zarrs-bindings/target/ diff --git a/changes/4181.feature.md b/changes/4181.feature.md index c28fbf48ff..26ca023d35 100644 --- a/changes/4181.feature.md +++ b/changes/4181.feature.md @@ -3,8 +3,7 @@ preserves existing behavior on every store and format. Pass `engine="zarrista"` (requires the `zarrista` package) to serve Zarr v3 arrays on local-filesystem, obstore, or icechunk storage through the Rust `zarrs` -implementation. The experimental `zarr.crud` and `zarr.zarrs` modules and the -bundled `zarrs-bindings` crate are removed in favor of this interface. +implementation. As part of this rewiring, writing a single field of a structured dtype (e.g. `z["x"] = ...`) now preserves the values of the array's other fields; it @@ -16,4 +15,7 @@ indices, with the requested elements picked out of that box afterwards. For sparse selections this can transfer more data from storage than before, and with the `array.read_missing_chunks` config set to `False`, a missing chunk that lies within the bounding box will raise even if no selected element -falls inside that chunk. +falls inside that chunk. Relatedly, a partial (non-full-box) write now performs +an internal read of the bounding box before writing it back, so with +`array.read_missing_chunks` set to `False` such a write raises +`ChunkNotFoundError` when it covers an uninitialized chunk. diff --git a/docs/api/zarr/abc/engine.md b/docs/api/zarr/abc/engine.md new file mode 100644 index 0000000000..fe38c62f40 --- /dev/null +++ b/docs/api/zarr/abc/engine.md @@ -0,0 +1,5 @@ +--- +title: engine +--- + +::: zarr.abc.engine diff --git a/docs/api/zarr/zarrista.md b/docs/api/zarr/zarrista.md new file mode 100644 index 0000000000..faf749495f --- /dev/null +++ b/docs/api/zarr/zarrista.md @@ -0,0 +1,5 @@ +--- +title: zarrista +--- + +::: zarr.zarrista diff --git a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md index fcb34d3629..f56c7e0859 100644 --- a/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md +++ b/docs/superpowers/specs/2026-07-22-array-engine-protocol-design.md @@ -100,7 +100,7 @@ class AsyncArrayEngine(Protocol): class AsyncHierarchyEngine(Protocol): def array_engine( - self, path: str, metadata: ArrayMetadata + self, path: str, metadata: ArrayMetadata, config: ArrayConfig | None = None ) -> AsyncArrayEngine: ... ``` @@ -223,8 +223,9 @@ Contract details: numpy via Zarrista's upcoming numpy export (a copy for now — confirmed planned); `MaskedTensor`/`MaskedVariableArray` unsupported (zarr-python has no masked dtype) — raise. -- Registered under the name `"zarrista"` on import of `zarr.zarrista`; import - errors cleanly when `zarrista` is not installed. +- Selected by the engine resolver under the name `"zarrista"`, which lazily + imports `zarr.zarrista` only when that name is resolved; the import errors + cleanly when `zarrista` is not installed. - Dependency: optional dependency group `zarrista`, git-pinned to a Zarrista `main` commit until its next release, then `>=` that release. diff --git a/mkdocs.yml b/mkdocs.yml index 7a4bfa35ef..37f363b02f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -48,10 +48,12 @@ nav: - api/zarr/registry.md - api/zarr/storage.md - api/zarr/experimental.md + - api/zarr/zarrista.md - ABC: - api/zarr/abc/index.md - api/zarr/abc/buffer.md - api/zarr/abc/codec.md + - api/zarr/abc/engine.md - api/zarr/abc/numcodec.md - api/zarr/abc/metadata.md - api/zarr/abc/store.md From 1a30404148d339fa7fa76c969e99b42c71c631d2 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 22 Jul 2026 23:44:20 +0200 Subject: [PATCH 67/75] fix(engine): keep device buffers in-namespace on scalar reads and squeeze widen Two residual host-coercion sites a full-path sweep found: - `_finalize_result` scalarize path used `np.asarray(result)[()]`, so an all-integer basic read (`get_basic_selection((1, 2))` / `getitem((1, 2))`) on a device engine raised `TypeError`. It now extracts the 0-d element in the result's own namespace unless the result is already numpy. - `_widen_value_for_squeeze` used `np.asarray(value)`, so an orthogonal write dropping an integer axis (`oindex[1, np.array([0, 2])] = device_value`) raised. It now checks `getattr(value, "ndim", 0)` and indexes the value in its own namespace. Adds three device-buffer regressions (sync + async scalar reads, oindex int+array write) that fail before and pass after. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 15 ++++-- tests/engine/test_device_buffer_facade.py | 62 +++++++++++++++++++++++ 2 files changed, 74 insertions(+), 3 deletions(-) diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 801a9daba0..1851ca01a4 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -5594,7 +5594,13 @@ def _finalize_result( if scalarize and result.shape == (): # basic indexing collapses to a scalar (matches the historical # `out_buffer.as_scalar()` return for all-integer basic selections) - return cast("NDArrayLikeOrScalar", np.asarray(result)[()]) + if isinstance(result, np.ndarray): + return cast("NDArrayLikeOrScalar", np.asarray(result)[()]) + # a device buffer (cupy/torch): extract the 0-d element in its own + # namespace rather than coercing to host via `np.asarray`, consistent + # with the value the orthogonal/fancy routes return + device_result: Any = result + return cast("NDArrayLikeOrScalar", device_result[()]) return result @@ -6045,11 +6051,14 @@ def _widen_value_for_squeeze( the box, matching `zarr.core.indexing.oindex_set`. Scalars and 0-d values are returned unchanged (they broadcast on their own). """ - if squeeze_axes_ and not np.isscalar(value) and np.asarray(value).ndim > 0: + if squeeze_axes_ and not np.isscalar(value) and getattr(value, "ndim", 0) > 0: value_selection: list[Any] = [slice(None)] * ndim for ax in squeeze_axes_: value_selection[ax] = np.newaxis - return cast("NDArrayLike", np.asarray(value)[tuple(value_selection)]) + # index the value in its own array namespace so a device buffer + # (cupy/torch) is widened on-device instead of coerced to host + indexed: Any = value + return cast("NDArrayLike", indexed[tuple(value_selection)]) return value diff --git a/tests/engine/test_device_buffer_facade.py b/tests/engine/test_device_buffer_facade.py index 7d97495789..12b2718327 100644 --- a/tests/engine/test_device_buffer_facade.py +++ b/tests/engine/test_device_buffer_facade.py @@ -117,6 +117,30 @@ def with_metadata(self, metadata: ArrayMetadata) -> _DeviceEngine: return self +class _AsyncDeviceEngine: + """Async mirror of `_DeviceEngine` for the `AsyncArray` data path.""" + + def __init__(self, data: npt.NDArray[Any]) -> None: + self._data = data + + async def read_selection( + self, selection: Region, *, prototype: BufferPrototype + ) -> _DeviceArray: + return _DeviceArray(self._data[_region_to_index(selection)].copy()) + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + arr: object = value.as_ndarray_like() + assert isinstance(arr, _DeviceArray), ( + "facade coerced the box off the device namespace before writing" + ) + self._data[_region_to_index(selection)] = arr._a + + def with_metadata(self, metadata: ArrayMetadata) -> _AsyncDeviceEngine: + return self + + def _array_on_device_engine() -> tuple[zarr.Array[Any], _DeviceEngine]: z = zarr.create_array(MemoryStore(), shape=(8,), chunks=(4,), dtype="int64") engine = _DeviceEngine(np.zeros(8, dtype="int64")) @@ -125,6 +149,13 @@ def _array_on_device_engine() -> tuple[zarr.Array[Any], _DeviceEngine]: return z, engine +def _array_2d_on_device_engine() -> tuple[zarr.Array[Any], _DeviceEngine]: + z = zarr.create_array(MemoryStore(), shape=(4, 4), chunks=(4, 4), dtype="int64") + engine = _DeviceEngine(np.arange(16, dtype="int64").reshape(4, 4)) + z._engine = engine # type: ignore[assignment] + return z, engine + + def test_identity_setitem_keeps_device_buffer() -> None: # full-box write: `_prepare_set_selection` broadcasts the value into the box # without a read; it must not `np.asarray` the device value. @@ -156,6 +187,37 @@ def test_strided_getitem_keeps_device_buffer() -> None: np.testing.assert_array_equal(result._a, np.arange(8, dtype="int64")[::2]) +def test_basic_selection_scalar_read_keeps_device_buffer() -> None: + # all-integer basic read scalarizes a 0-d result; `_finalize_result` must + # extract the element in the device namespace, not via `np.asarray(result)[()]`. + z, engine = _array_2d_on_device_engine() + result: object = z.get_basic_selection((1, 2)) + assert isinstance(result, _DeviceArray), "scalar read coerced off the device namespace" + np.testing.assert_array_equal(result._a, engine._data[1, 2]) + + +async def test_async_getitem_scalar_read_keeps_device_buffer() -> None: + z = zarr.create_array(MemoryStore(), shape=(4, 4), chunks=(4, 4), dtype="int64") + aa = z.async_array + engine = _AsyncDeviceEngine(np.arange(16, dtype="int64").reshape(4, 4)) + object.__setattr__(aa, "engine", engine) + result: object = await aa.getitem((1, 2)) + assert isinstance(result, _DeviceArray), "async scalar read coerced off the device namespace" + np.testing.assert_array_equal(result._a, engine._data[1, 2]) + + +def test_oindex_set_integer_and_array_axis_keeps_device_buffer() -> None: + # orthogonal write with a dropped integer axis widens the value with a + # `np.newaxis`; `_widen_value_for_squeeze` must index the value in its own + # namespace instead of `np.asarray(value)[...]`. + z, engine = _array_2d_on_device_engine() + value = _DeviceArray(np.array([10, 20], dtype="int64")) + z.oindex[1, np.array([0, 2])] = value + expected = np.arange(16, dtype="int64").reshape(4, 4) + expected[1, [0, 2]] = [10, 20] + np.testing.assert_array_equal(engine._data, expected) + + def test_facade_never_coerces_device_buffer_to_host() -> None: # Belt-and-braces: a bare `np.asarray` on the stand-in must raise, proving the # tests above would trip any host coercion the facade performed. From 2bde8f69d7932bb541853f8ac0fa055d510bb615 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 08:03:30 +0200 Subject: [PATCH 68/75] fix(engine): route numpy scalar results through the host scalar-extraction path The device-buffer scalarize branch guarded with `isinstance(result, np.ndarray)`, but a vlen/fixed-bytes scalar read produces a numpy scalar (`np.bytes_`, an `np.generic`, not an `ndarray`); it fell to the device path `result[()]`, and `np.bytes_[()]` raises "byte indices must be integers or slices, not tuple" (surfaced by the `test_basic_indexing` hypothesis suite). Widen the guard to `(np.ndarray, np.generic)` so every numpy result takes the bit-identical `np.asarray(...)[()]` path; only genuine device buffers (neither type) are indexed in their own namespace. Adds a deterministic regression (`get_basic_selection(0)` on a 1-d `S4` array). Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 12 +++++++----- tests/engine/test_device_buffer_facade.py | 15 +++++++++++++++ 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index 1851ca01a4..a77ceb2bd0 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -5593,12 +5593,14 @@ def _finalize_result( return out.as_ndarray_like() if scalarize and result.shape == (): # basic indexing collapses to a scalar (matches the historical - # `out_buffer.as_scalar()` return for all-integer basic selections) - if isinstance(result, np.ndarray): + # `out_buffer.as_scalar()` return for all-integer basic selections). + # Any numpy result -- a 0-d `ndarray` or an already-collapsed numpy + # scalar (`np.generic`, e.g. `np.bytes_` from a vlen/fixed-bytes read) -- + # goes through `np.asarray(...)[()]`, bit-identical to the old behaviour; + # only a genuine device buffer (cupy/torch), which is neither, is indexed + # in its own namespace to avoid a host coercion. + if isinstance(result, (np.ndarray, np.generic)): return cast("NDArrayLikeOrScalar", np.asarray(result)[()]) - # a device buffer (cupy/torch): extract the 0-d element in its own - # namespace rather than coercing to host via `np.asarray`, consistent - # with the value the orthogonal/fancy routes return device_result: Any = result return cast("NDArrayLikeOrScalar", device_result[()]) return result diff --git a/tests/engine/test_device_buffer_facade.py b/tests/engine/test_device_buffer_facade.py index 12b2718327..a86e841419 100644 --- a/tests/engine/test_device_buffer_facade.py +++ b/tests/engine/test_device_buffer_facade.py @@ -218,6 +218,21 @@ def test_oindex_set_integer_and_array_axis_keeps_device_buffer() -> None: np.testing.assert_array_equal(engine._data, expected) +@pytest.mark.filterwarnings("ignore::zarr.core.dtype.common.UnstableSpecificationWarning") +def test_scalar_bytes_read_returns_numpy_scalar() -> None: + # A fixed/vlen-bytes scalar read produces a numpy scalar (`np.bytes_`), not a + # 0-d ndarray; the device-namespace `result[()]` branch must not swallow it + # (`np.bytes_[()]` raises "byte indices must be integers"). numpy scalars go + # through the `np.asarray` path, staying bit-identical to the historical + # `as_scalar()` return. (Distilled from a `test_basic_indexing` hypothesis + # failure; uses the real default engine, no device stand-in.) + z = zarr.create_array(MemoryStore(), shape=(1,), chunks=(1,), dtype="S4") + z[:] = np.array([b"ab"], dtype="S4") + result = z.get_basic_selection(0) + assert isinstance(result, np.bytes_) + assert result == b"ab" + + def test_facade_never_coerces_device_buffer_to_host() -> None: # Belt-and-braces: a bare `np.asarray` on the stand-in must raise, proving the # tests above would trip any host coercion the facade performed. From 66b24927adac66d3ef4a55769ed7d460584f1d43 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 09:13:49 +0200 Subject: [PATCH 69/75] fix(engine): return writable arrays from identity engine reads The identity-read fast path in `_finish_get_selection` returned the engine's buffer unchanged. An engine may expose read-only memory (e.g. `zarrista` wraps Rust-owned bytes that `np.asarray` sees as non-writable), so a full-box read via engine="zarrista" could return a read-only array, breaking zarr-python's long-standing writable-read guarantee. Copy a numpy-visible, non-writable result on that path; leave device buffers (no `.flags`) untouched so they are never forced onto the host. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/core/array.py | 12 ++++++- tests/engine/test_asyncarray_wiring.py | 43 ++++++++++++++++++++++++++ tests/zarrista/test_engine.py | 14 +++++++++ 3 files changed, 68 insertions(+), 1 deletion(-) diff --git a/src/zarr/core/array.py b/src/zarr/core/array.py index a77ceb2bd0..7dfd9d65ad 100644 --- a/src/zarr/core/array.py +++ b/src/zarr/core/array.py @@ -5681,7 +5681,17 @@ def _finish_get_selection( if not fields and _is_identity_read(post_index, region.shape): # a full-box, step-1 read is exactly the engine result; return it # unchanged so a custom `NDArrayLike` type (e.g. GPU/torch buffers) - # survives instead of being coerced to numpy by `np.asarray` + # survives instead of being coerced to numpy by `np.asarray`. + # zarr-python reads have always returned writable arrays; the non-identity + # paths below copy (via `astype(copy=True)`), but this fast path hands the + # engine buffer straight back. An engine may expose read-only memory (e.g. + # `zarrista` wraps Rust-owned bytes that `np.asarray` sees as + # non-writable), so copy a numpy-visible, non-writable result to restore + # the writability guarantee. Device buffers (cupy/torch) have no `.flags` + # and are left untouched, so this never forces them onto the host. + flags = getattr(raw, "flags", None) + if flags is not None and not flags.writeable: + raw = raw.copy() return _finalize_result(raw, out, scalarize=scalarize) # Stay in the engine result's own array namespace: field selection and # `apply_post_index` are plain indexing (which cupy/torch implement), so a diff --git a/tests/engine/test_asyncarray_wiring.py b/tests/engine/test_asyncarray_wiring.py index 5725f57065..dd234acbc7 100644 --- a/tests/engine/test_asyncarray_wiring.py +++ b/tests/engine/test_asyncarray_wiring.py @@ -37,6 +37,49 @@ def with_metadata(self, metadata: ArrayMetadata) -> _SpyEngine: return _SpyEngine(self.inner.with_metadata(metadata)) +class _ReadOnlyEngine: + """Wraps a real engine but returns a read-only numpy view from reads. + + Mimics an engine (e.g. `zarrista`) that exposes foreign-owned memory as a + non-writable numpy array. The facade must copy such a result so a read + still yields a writable array. + """ + + def __init__(self, inner: DefaultAsyncArrayEngine) -> None: + self.inner = inner + + async def read_selection(self, selection: Region, *, prototype: BufferPrototype) -> NDArrayLike: + result = await self.inner.read_selection(selection, prototype=prototype) + view = np.asarray(result) + view.flags.writeable = False + return cast("NDArrayLike", view) + + async def write_selection( + self, selection: Region, value: NDBuffer, *, prototype: BufferPrototype + ) -> None: + return await self.inner.write_selection(selection, value, prototype=prototype) + + def with_metadata(self, metadata: ArrayMetadata) -> _ReadOnlyEngine: + return _ReadOnlyEngine(self.inner.with_metadata(metadata)) + + +async def test_read_only_engine_result_is_copied_to_writable() -> None: + # Regression: the identity-read fast path returned the engine buffer + # unchanged. When an engine returns non-writable numpy memory, the facade + # must copy it so reads keep zarr-python's writable-array guarantee. + z = zarr.create_array(MemoryStore(), shape=(6,), chunks=(3,), dtype="int16") + aa = z.async_array + inner = DefaultAsyncArrayEngine( + store_path=aa.store_path, metadata=aa.metadata, config=aa.config + ) + object.__setattr__(aa, "engine", _ReadOnlyEngine(inner)) + await aa.setitem(slice(None), np.arange(6, dtype="int16")) + + result = np.asarray(await aa.getitem(slice(None))) + assert result.flags.writeable + np.testing.assert_array_equal(result, np.arange(6, dtype="int16")) + + async def test_asyncarray_routes_io_through_engine() -> None: # NOTE: the async variant of the spy test. `Array` (the sync facade) now # resolves and calls its own sync engine (see tests/engine/test_sync_path.py); diff --git a/tests/zarrista/test_engine.py b/tests/zarrista/test_engine.py index 148899525a..1879b51afc 100644 --- a/tests/zarrista/test_engine.py +++ b/tests/zarrista/test_engine.py @@ -56,6 +56,20 @@ def test_zarrista_engine_edge_chunk_full_write(tmp_path: Path) -> None: ) +def test_zarrista_reads_are_writable(tmp_path: Path) -> None: + # zarrista's `Tensor` wraps Rust-owned memory that `np.asarray` exposes + # read-only. zarr-python reads have always returned writable arrays, so the + # facade must copy such a result -- both on the full-box identity path and + # on a partial (bounding-box) read. + _make(tmp_path) + ze = zarr.open_array(LocalStore(tmp_path), engine="zarrista") + + full = np.asarray(ze[:, :]) + assert full.flags.writeable + partial = np.asarray(ze[2:7, 1:5]) + assert partial.flags.writeable + + def test_zarrista_rejects_v2(tmp_path: Path) -> None: zarr.create_array(LocalStore(tmp_path), shape=(4,), chunks=(2,), dtype="int8", zarr_format=2) with pytest.raises(UnsupportedEngineError, match="v3"): From fdb4e5ecdfdde76099b8e3755bdba195ba5337d6 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 11:12:12 +0200 Subject: [PATCH 70/75] feat(engine): reject read_missing_chunks=False for the zarrista engine The zarrista engines ignore zarr-python's `ArrayConfig`, so `read_missing_chunks=False` would silently fill missing chunks with the fill value where the default engine raises `ChunkNotFoundError`. Rather than downgrade the semantics silently, `array_engine` (sync and async) now raises `UnsupportedEngineError` when handed a config with `read_missing_chunks=False`, pointing at the default engine. Other `ArrayConfig` fields (e.g. `order`) only affect in-memory layout the facade normalizes, so they remain safely ignored. Assisted-by: ClaudeCode:claude-opus-4-8 --- src/zarr/zarrista/_engine.py | 42 +++++++++++++++++++++++++++++------ tests/zarrista/test_engine.py | 27 ++++++++++++++++++++++ 2 files changed, 62 insertions(+), 7 deletions(-) diff --git a/src/zarr/zarrista/_engine.py b/src/zarr/zarrista/_engine.py index 51aa06f7c3..4e25549f31 100644 --- a/src/zarr/zarrista/_engine.py +++ b/src/zarr/zarrista/_engine.py @@ -39,6 +39,26 @@ def _require_v3(metadata: ArrayMetadata) -> ArrayMetadataV3: return cast("ArrayMetadataV3", metadata.to_dict()) +def _reject_unenforceable_config(config: ArrayConfig | None) -> None: + """Reject an `ArrayConfig` the zarrista engine cannot honour. + + The zarrista engine owns its own codec options and does not consult + zarr-python's `ArrayConfig`. Most fields (e.g. `order`) only affect the + in-memory layout of the returned array, which the facade normalizes, so + ignoring them is safe. `read_missing_chunks=False`, however, changes + *semantics*: the default engine raises `ChunkNotFoundError` for a missing + chunk, whereas zarrista silently fills it with the fill value. Per the + project's fail-loud rule, refuse rather than silently downgrade to + fill-value reads. + """ + if config is not None and not config.read_missing_chunks: + raise UnsupportedEngineError( + "the zarrista engine cannot enforce read_missing_chunks=False " + "(it fills missing chunks with the fill value instead of raising); " + "use the default engine to enforce this setting" + ) + + def _region_to_selection(region: Region) -> tuple[slice, ...]: return tuple(slice(s, e) for s, e in zip(region.start, region.end_exclusive, strict=True)) @@ -142,11 +162,15 @@ def array_engine( """Mint a sync array engine bound to `path`/`metadata`. `config` is accepted for protocol conformance with `HierarchyEngine` - but is unused: zarrista owns its own codec options and does not read - zarr-python's `ArrayConfig` (e.g. `order`, `read_missing_chunks`). + and is otherwise unused -- zarrista owns its own codec options and does + not read zarr-python's `ArrayConfig` (e.g. `order`) -- with one + exception: `config.read_missing_chunks=False` is rejected with an + `UnsupportedEngineError`, since zarrista cannot enforce it and would + silently fill missing chunks instead of raising. """ import zarrista + _reject_unenforceable_config(config) return ZarristaEngine( zarrista.Array.from_metadata(_require_v3(metadata), self._zstore, "/" + path.strip("/")) ) @@ -244,11 +268,15 @@ def array_engine( """Mint an async array engine bound to `path`/`metadata`. `config` is accepted for protocol conformance with - `AsyncHierarchyEngine` but is unused: zarrista owns its own codec - options and does not read zarr-python's `ArrayConfig` (e.g. `order`, - `read_missing_chunks`). `metadata` is validated as Zarr v3 eagerly - (cheap, and lets an unsupported-format error surface immediately); - the store itself is only translated lazily, on first I/O. + `AsyncHierarchyEngine` and is otherwise unused -- zarrista owns its own + codec options and does not read zarr-python's `ArrayConfig` (e.g. + `order`) -- with one exception: `config.read_missing_chunks=False` is + rejected with an `UnsupportedEngineError`, since zarrista cannot enforce + it and would silently fill missing chunks instead of raising. `metadata` + is validated as Zarr v3 eagerly (cheap, and lets an unsupported-format + error surface immediately); the store itself is only translated lazily, + on first I/O. """ + _reject_unenforceable_config(config) _require_v3(metadata) return ZarristaAsyncEngine(self._zarr_store, "/" + path.strip("/"), metadata) diff --git a/tests/zarrista/test_engine.py b/tests/zarrista/test_engine.py index 1879b51afc..1a590c6757 100644 --- a/tests/zarrista/test_engine.py +++ b/tests/zarrista/test_engine.py @@ -88,6 +88,33 @@ def test_zarrista_vlen_read_not_implemented(tmp_path: Path) -> None: ze[:] +def test_zarrista_sync_rejects_read_missing_chunks_false(tmp_path: Path) -> None: + # The zarrista engine cannot enforce read_missing_chunks=False (it fills + # missing chunks instead of raising), so minting a sync engine with that + # config must fail loudly rather than silently downgrade the semantics. + from zarr.core.array_spec import ArrayConfig + from zarr.zarrista._engine import ZarristaHierarchyEngine + + z = _make(tmp_path) + config = ArrayConfig(order="C", write_empty_chunks=False, read_missing_chunks=False) + hierarchy = ZarristaHierarchyEngine(LocalStore(tmp_path)) + with pytest.raises(UnsupportedEngineError, match="read_missing_chunks=False"): + hierarchy.array_engine("", z.metadata, config) + + +async def test_zarrista_async_rejects_read_missing_chunks_false(tmp_path: Path) -> None: + # Same fail-loud contract as the sync engine, exercised on the async + # hierarchy engine's `array_engine` factory. + from zarr.core.array_spec import ArrayConfig + from zarr.zarrista._engine import ZarristaAsyncHierarchyEngine + + z = _make(tmp_path) + config = ArrayConfig(order="C", write_empty_chunks=False, read_missing_chunks=False) + hierarchy = ZarristaAsyncHierarchyEngine(LocalStore(tmp_path)) + with pytest.raises(UnsupportedEngineError, match="read_missing_chunks=False"): + hierarchy.array_engine("", z.metadata, config) + + async def test_zarrista_async_engine_read_write_combinations(tmp_path: Path) -> None: # exercises `ZarristaAsyncEngine` directly (over an obstore-backed # `ObjectStore`), rather than through the sync `Array`/`ZarristaEngine` From a0c211fdfd9ce82dff71d285bc528e132571b61c Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 11:16:43 +0200 Subject: [PATCH 71/75] feat(engine): add zarr.list_engines() discoverability API `zarr.list_engines()` returns the sorted names of the built-in array engines (`["default", "zarrista"]`), derived from the `EngineName` literal that `_hierarchy_factory` already dispatches on -- so the set of names lives in one place. `"zarrista"` requires the optional `zarrista` package. Re-exported from the top level and noted in the engine changelog fragment. Assisted-by: ClaudeCode:claude-opus-4-8 --- changes/4181.feature.md | 3 ++- src/zarr/__init__.py | 2 ++ src/zarr/core/engine/__init__.py | 2 ++ src/zarr/core/engine/_resolve.py | 18 +++++++++++++++++- tests/engine/test_resolve.py | 7 +++++++ 5 files changed, 30 insertions(+), 2 deletions(-) diff --git a/changes/4181.feature.md b/changes/4181.feature.md index 26ca023d35..ebaf18c11d 100644 --- a/changes/4181.feature.md +++ b/changes/4181.feature.md @@ -3,7 +3,8 @@ preserves existing behavior on every store and format. Pass `engine="zarrista"` (requires the `zarrista` package) to serve Zarr v3 arrays on local-filesystem, obstore, or icechunk storage through the Rust `zarrs` -implementation. +implementation. Call `zarr.list_engines()` to discover the available engine +names. As part of this rewiring, writing a single field of a structured dtype (e.g. `z["x"] = ...`) now preserves the values of the array's other fields; it diff --git a/src/zarr/__init__.py b/src/zarr/__init__.py index cdf3840c3b..261fb6cfdf 100644 --- a/src/zarr/__init__.py +++ b/src/zarr/__init__.py @@ -36,6 +36,7 @@ ) from zarr.core.array import Array, AsyncArray from zarr.core.config import config +from zarr.core.engine import list_engines from zarr.core.group import AsyncGroup, Group # in case setuptools scm screw up and find version to be 0.0.0 @@ -164,6 +165,7 @@ def set_format(log_format: str) -> None: "full", "full_like", "group", + "list_engines", "load", "ones", "ones_like", diff --git a/src/zarr/core/engine/__init__.py b/src/zarr/core/engine/__init__.py index 23b626aa29..ade232b6c3 100644 --- a/src/zarr/core/engine/__init__.py +++ b/src/zarr/core/engine/__init__.py @@ -15,6 +15,7 @@ from zarr.core.engine._resolve import ( EngineName, classify_engine_arg, + list_engines, resolve_async_engine, resolve_sync_engine, route_sync_engine_arg, @@ -28,6 +29,7 @@ "EngineName", "apply_post_index", "classify_engine_arg", + "list_engines", "normalize_basic", "normalize_coordinate", "normalize_orthogonal", diff --git a/src/zarr/core/engine/_resolve.py b/src/zarr/core/engine/_resolve.py index 3330e0ccd7..7b2f3bbe0c 100644 --- a/src/zarr/core/engine/_resolve.py +++ b/src/zarr/core/engine/_resolve.py @@ -5,7 +5,7 @@ import contextlib import inspect import weakref -from typing import TYPE_CHECKING, Literal +from typing import TYPE_CHECKING, Literal, get_args from zarr.core.engine._default import ( DefaultAsyncHierarchyEngine, @@ -28,6 +28,7 @@ __all__ = [ "EngineName", "classify_engine_arg", + "list_engines", "resolve_async_engine", "resolve_sync_engine", "route_sync_engine_arg", @@ -36,6 +37,21 @@ EngineName = Literal["default", "zarrista"] +def list_engines() -> list[str]: + """Return the sorted names of the built-in array engines. + + Any of these names can be passed as the `engine=` argument to + `zarr.open_array`, `zarr.create_array`, and the other array entry points to + select the data-path engine backing the array. + + `"zarrista"` additionally requires the optional `zarrista` package to be + installed; without it, resolving that engine raises an `ImportError`. + """ + # `EngineName` is the single source of truth for known engine names -- + # `_hierarchy_factory` dispatches on exactly these literals. + return sorted(get_args(EngineName)) + + def classify_engine_arg(engine: object) -> Literal["name", "sync", "async"]: """Classify an `engine=` argument as a name, a sync instance, or an async instance. diff --git a/tests/engine/test_resolve.py b/tests/engine/test_resolve.py index 73f09dc8ae..30bda6c694 100644 --- a/tests/engine/test_resolve.py +++ b/tests/engine/test_resolve.py @@ -20,6 +20,13 @@ def _array() -> zarr.Array[Any]: return zarr.create_array(MemoryStore(), shape=(4,), chunks=(2,), dtype="int8") +def test_list_engines() -> None: + # `list_engines` reports exactly the known engine names, sorted, and is + # re-exported at the top level for discoverability. + assert zarr.list_engines() == ["default", "zarrista"] + assert callable(zarr.list_engines) + + def test_resolution_combinations() -> None: z = _array() store = z.store From 8686b3f52305c6de1ae122a82e335339bd67212f Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 11:29:59 +0200 Subject: [PATCH 72/75] docs(engine): add open_with_engine runnable example Port the old tip's open-with-backend example to our engine API: select an engine per call via `engine="default"`/`engine="zarrista"` on `zarr.open_array`/`zarr.create_array`, discover names via `zarr.list_engines()`, and read one on-disk array back through each engine asserting byte-for-byte equality. The zarrista portion is guarded behind a real `import zarrista` probe (importing `zarr.zarrista` alone succeeds even without the package), so the script runs with or without the `zarrista` group. Wired the docs page into mkdocs nav. Assisted-by: ClaudeCode:claude-opus-4-8 --- docs/user-guide/examples/open_with_engine.md | 7 + examples/open_with_engine/README.md | 52 +++++++ examples/open_with_engine/open_with_engine.py | 129 ++++++++++++++++++ mkdocs.yml | 1 + 4 files changed, 189 insertions(+) create mode 100644 docs/user-guide/examples/open_with_engine.md create mode 100644 examples/open_with_engine/README.md create mode 100644 examples/open_with_engine/open_with_engine.py diff --git a/docs/user-guide/examples/open_with_engine.md b/docs/user-guide/examples/open_with_engine.md new file mode 100644 index 0000000000..30848612c9 --- /dev/null +++ b/docs/user-guide/examples/open_with_engine.md @@ -0,0 +1,7 @@ +--8<-- "examples/open_with_engine/README.md" + +## Source Code + +```python exec="false" reason="pymdownx snippet include directive, not python source" +--8<-- "examples/open_with_engine/open_with_engine.py" +``` diff --git a/examples/open_with_engine/README.md b/examples/open_with_engine/README.md new file mode 100644 index 0000000000..0209babcb8 --- /dev/null +++ b/examples/open_with_engine/README.md @@ -0,0 +1,52 @@ +# Open With a Different Engine Example + +This example demonstrates how to open the **same array data with a different +backend** -- what zarr-python calls an *engine*. + +An engine is purely an *execution* setting: it selects which compute backend +reads and writes an array's chunks. The bytes on disk are identical regardless +of the engine, so the same Zarr array can be driven by a different backend +without rewriting any data. + +The example shows how to: + +- Discover the available engines with `zarr.list_engines()`. +- Inspect the engine an array is using via its public `array.engine` property. +- Write one array once and read it back through several engines, asserting the + results are byte-for-byte identical (`numpy.testing.assert_array_equal`). +- Select an engine **per call** via the `engine=` kwarg on `zarr.open_array` + and `zarr.create_array`. +- Use the `"default"` engine, which works on any store and format. +- Use the `"zarrista"` (Rust-backed) engine, which serves Zarr v3 arrays on a + `LocalStore` or an obstore-backed `ObjectStore` (the example guards this so it + still runs if the package is absent). + +## Available engines + +| Engine | Backend | Stores | Notes | +| ------------ | ----------------- | ----------------------------------- | ------------------------------- | +| `"default"` | built-in Python | any | used when `engine=` is omitted | +| `"zarrista"` | Rust (`zarrista`) | `LocalStore`, obstore `ObjectStore` | requires the `zarrista` package | + +## Running the Example + +This example demonstrates an **unreleased** feature and the `"zarrista"` engine +depends on the optional `zarrista` package (pulled in through the `zarrista` +dependency group). It therefore does **not** carry a PEP 723 inline-dependency +header like some other examples: such a header would install a zarr without this +feature and fail at runtime. + +Run it from a checkout of this branch: + +```bash +uv run python examples/open_with_engine/open_with_engine.py +``` + +To exercise the Rust-backed engine, sync the `zarrista` dependency group: + +```bash +uv run --group zarrista python examples/open_with_engine/open_with_engine.py +``` + +If `zarrista` is not installed, the example still runs and demonstrates the +default engine; the `zarrista` portion is skipped. diff --git a/examples/open_with_engine/open_with_engine.py b/examples/open_with_engine/open_with_engine.py new file mode 100644 index 0000000000..f7bdfbb378 --- /dev/null +++ b/examples/open_with_engine/open_with_engine.py @@ -0,0 +1,129 @@ +# NOTE on dependencies / how to run +# ---------------------------------- +# Unlike some other examples in this directory, this one does NOT carry a PEP 723 +# `# /// script` inline-dependency header. +# +# The feature it demonstrates -- selecting a per-array execution "engine" -- is +# unreleased: it is not present in any published release. The `"zarrista"` engine +# additionally requires the `zarrista` package, which this repo pulls in through +# the `zarrista` dependency *group* (something a single inline `dependencies` +# pin cannot express cleanly). A PEP 723 header pinning a published zarr would +# therefore silently install a zarr WITHOUT this feature and fail at runtime, so +# we deliberately omit it. +# +# To run this example from a checkout of this branch: +# +# uv run python examples/open_with_engine/open_with_engine.py +# +# and, with the Rust-backed engine available: +# +# uv run --group zarrista python examples/open_with_engine/open_with_engine.py +# +# The zarrista portion is guarded: if the `zarrista` engine cannot be imported, +# the example still runs and demonstrates the default engine. + +""" +Open the same array data with a different backend ("engine"). + +zarr-python routes an array's data I/O through a selectable *engine*. The engine +is purely an *execution* setting: it chooses which compute backend reads and +writes the array's chunks. The bytes on disk are identical regardless of the +engine -- the same Zarr array, just a different machine doing the work. + +Engines demonstrated here: + +- `"default"` -- the built-in engine (used when `engine=` is omitted). Works on + every store and format. +- `"zarrista"` -- a Rust-backed engine (via the `zarrista` package) that serves + Zarr v3 arrays on a `LocalStore` or an obstore-backed `ObjectStore`. Guarded: + skipped if the package is not installed. + +We write one array once, then open and read it back through each engine and +assert the results are byte-for-byte identical. +""" + +import tempfile +from pathlib import Path + +import numpy as np + +import zarr +from zarr.storage import LocalStore + + +def zarrista_available() -> bool: + """Report whether the optional Rust-backed `"zarrista"` engine can be used. + + The `"zarrista"` engine imports the `zarrista` package lazily, so we probe + that package directly: importing `zarr.zarrista` alone succeeds even when the + package is absent (the actual `ImportError` would only surface on first I/O). + """ + try: + import zarrista # noqa: F401 + except ImportError: + return False + return True + + +def main() -> None: + # Discover which engines exist. `zarr.list_engines()` returns the built-in + # engine names; `"zarrista"` additionally requires the `zarrista` package. + print("available engines:", zarr.list_engines()) + + # zarrista ingests a LocalStore (used here, on a temp directory) or an + # obstore-backed ObjectStore. We keep a single store so every engine reads + # the exact same bytes off the same disk. + with tempfile.TemporaryDirectory() as tmp: + store = LocalStore(Path(tmp) / "store") + + # The data we will write once and read back through several engines. + data = np.arange(8 * 8, dtype="uint16").reshape(8, 8) + + # --- Write the array once, with the default engine. ----------------- + # No engine= here, so this uses the "default" engine. + source = zarr.create_array( + store=store, name="a", shape=(8, 8), chunks=(4, 4), dtype="uint16" + ) + source[:] = data + + # --- 1. Default engine (the baseline). ------------------------------ + # Either omit engine= or pass engine="default"; both mean the built-in. + default = zarr.open_array(store=store, path="a", engine="default") + # `array.engine` is the resolved engine instance backing this array. + print("default engine:", type(default.engine).__name__) + np.testing.assert_array_equal(default[:], data) + # Basic indexing (ints and slices) is what engines route; check a slice. + np.testing.assert_array_equal(default[2:6, 1:5], data[2:6, 1:5]) + print("default (engine='default') : read back OK") + + # --- 2. zarrista engine (Rust), guarded behind availability. -------- + if zarrista_available(): + zst = zarr.open_array(store=store, path="a", engine="zarrista") + print("zarrista engine:", type(zst.engine).__name__) + np.testing.assert_array_equal(zst[:], data) + np.testing.assert_array_equal(zst[2:6, 1:5], data[2:6, 1:5]) + # Same bytes on disk, different compute backend: identical to default. + np.testing.assert_array_equal(zst[:], default[:]) + print("zarrista (engine='zarrista') : read back OK, equals default") + + # Writes also route through the engine. Create + write + read back + # entirely through zarrista, then confirm a *default* reader agrees. + written = zarr.create_array( + store=store, + name="b", + shape=(8, 8), + chunks=(4, 4), + dtype="uint16", + engine="zarrista", + ) + written[:] = data + np.testing.assert_array_equal(zarr.open_array(store=store, path="b")[:], data) + print("zarrista (write path) : round-trip OK, default reader agrees") + else: + print("zarrista : SKIPPED (package not installed)") + + print("\nAll engines returned identical data. Same bytes on disk, different backend.") + + +if __name__ == "__main__": + main() diff --git a/mkdocs.yml b/mkdocs.yml index 37f363b02f..6387c23b59 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -30,6 +30,7 @@ nav: - user-guide/glossary.md - Examples: - user-guide/examples/custom_dtype.md + - user-guide/examples/open_with_engine.md - user-guide/examples/rectilinear_chunks.ipynb - API Reference: - api/zarr/index.md From a0907f69e8fec72956cade474133d49c7f9b27fb Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 12:10:09 +0200 Subject: [PATCH 73/75] docs: post-mortem of zarr-python blockers for smooth engine integration Assisted-by: ClaudeCode:claude-fable-5 --- ...-python-blockers-for-engine-integration.md | 233 ++++++++++++++++++ 1 file changed, 233 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-23-zarr-python-blockers-for-engine-integration.md diff --git a/docs/superpowers/specs/2026-07-23-zarr-python-blockers-for-engine-integration.md b/docs/superpowers/specs/2026-07-23-zarr-python-blockers-for-engine-integration.md new file mode 100644 index 0000000000..2df8a14685 --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-zarr-python-blockers-for-engine-integration.md @@ -0,0 +1,233 @@ +# What zarr-python should change for engines to fit naturally + +Date: 2026-07-23 +Status: discussion draft +Context: post-mortem of the array-engine-protocol branch +(`claude/zarr-array-engine-protocol-239cfa`, spec: +`2026-07-22-array-engine-protocol-design.md`). The engine layer works and is +fully tested, but building it meant working *around* several patterns baked +into zarr-python rather than *with* them. This document names those patterns, +shows the concrete friction each caused on the branch, and proposes the +change that would have made — and would still make — the integration smooth. + +Ranked by how much friction each caused. + +## 1. The `Array` / `AsyncArray` dual-class architecture + +**The pattern.** `Array` is a wrapper holding an `AsyncArray`; every sync +operation historically round-tripped through `sync()` onto the loop thread +(22 `sync()` call sites remain in `array.py`). Every open constructs both +layers unconditionally. + +**The friction.** The engine concept is *exactly* a sync/async boundary — so +the class pair duplicates it. Consequences on the branch: + +- Every array conceptually needs **two** engines (sync for `Array`, async for + `AsyncArray`), but constructing both eagerly is impossible for zarrista: + its sync side translates only `LocalStore`, its async side only + obstore/icechunk — disjoint sets, so dual eager resolution fails on + *every* store. We papered over this twice: `Array._engine` is a lazily + resolved nullable cache slot, and `ZarristaAsyncEngine` defers store + translation to first I/O. Both are workarounds for "the wrapper class + forces two engines where one is ever used." +- `Array` is a dataclass, so a constructor keyword `engine=` collides with + the `engine` property (verified: it silently corrupts the field default) — + hence the awkward internal `engine_spec: InitVar` spelling. +- Sync entry points must thread the engine spec to *both* layers so sync and + async access agree — more plumbing per entry point. + +**The change.** Make the engine the *only* sync/async boundary. One `Array` +class parameterized by its engine; `.async_` / `.sync_` access becomes a +property of the engine binding, not a parallel class hierarchy. Short of +that: stop having `Array` eagerly construct an `AsyncArray` (peer classes +over shared `(store, path, metadata, config)` state, each resolving only its +own engine on demand). + +## 2. Creation/open API sprawl + +**The pattern.** Twelve `create_array`/`open_array`/`create`/`from_array`/ +`init_array` entry points across five files (`zarr.api.synchronous`, +`zarr.api.asynchronous`, `zarr.core.array`, `zarr.core.group`), each +duplicating a ~25-keyword signature. + +**The friction.** Adding one keyword (`engine=`) meant editing every entry +point and every intermediate hop (15 mentions in `asynchronous.py`, 24 in +`synchronous.py`). Task 7 was an entire plan task whose content was pure +signature threading, and the old branch tip independently hit a bug in +exactly this plumbing (`open_array`'s create-fallback dropping `engine`). +The still-open `open_array(config=...)`-ignored-on-existing-arrays gap is +the same disease: `AsyncArray.open` simply has no `config` parameter. + +**The change.** Collapse creation/open onto one internal path taking a +single spec object (an `ArrayParams`/builder holding shape, dtype, chunks, +codecs, config, engine, ...). Public functions become thin adapters that +construct the spec; the sync API is generated mechanically from the async +one. New cross-cutting parameters then land in exactly one place. + +## 3. `Store` hides the native resource + +**The pattern.** `zarr.abc.store.Store` is an async Python byte interface. +That is the right abstraction for Python codecs, but a native engine does +not want Python byte callbacks — it wants the *underlying handle*: a +filesystem root, an obstore instance, an icechunk session. + +**The friction.** `zarr.zarrista._translate` is isinstance-sniffing: +`LocalStore → FilesystemStore` via `.root`, `ObjectStore → .store`, +icechunk via a guarded import. Third-party stores cannot participate at +all; `MemoryStore` is fundamentally untranslatable (process-local Python +dict vs Rust address space) and needs a bespoke error message; and the +sync/async translatable sets being disjoint caused the lazy-resolution +contortions of §1. + +**The change.** A capability protocol on `Store` — e.g. +`def native_handle(self) -> NativeStoreDescriptor | None`, where the +descriptor is a small tagged union (`kind: "filesystem" | "obstore" | +"icechunk" | ...`, plus the handle). Stores opt in; engines match on the +descriptor instead of on zarr-python class identity; third-party stores +become translatable without `zarr.zarrista` knowing their types. A stronger +version: standardize object storage on obstore internally so "the handle" +exists for every remote store by construction. + +## 4. Runtime config has no single home + +**The pattern.** `ArrayConfig` (order, `read_missing_chunks`, ...) is parsed +per-array but threaded ad hoc; engines are bound to `(store, path, +metadata)` and got `config` bolted onto `HierarchyEngine.array_engine` +mid-project when the default engine turned out to need it (it owns +`read_missing_chunks` enforcement). + +**The friction.** The protocol churned after Kyle's review (the spec's +hierarchy snippet had to be amended); zarrista engines must now *reject* +config they cannot honor (`read_missing_chunks=False`) rather than share an +enforcement point; `with_config` silently dropped the engine until the +final review caught it, because config, metadata, and engine rebind through +three different mechanisms. + +**The change.** Bind arrays and engines to one immutable context object: +`ArrayContext(store, path, metadata, config)`. Engine construction takes +the context; every metadata/config change produces a successor context and +rebinding becomes one operation (`engine.with_context(ctx)`) instead of +`with_metadata` + config threading + per-site `_rebind_engine()` calls. + +## 5. Metadata mutation has three inconsistent successor patterns + +**The pattern.** `resize` mutates `metadata` in place on the same +`AsyncArray`; `with_config` constructs a fresh wrapper; `update_attributes` +mutates a shared dict *and* returns a new `Array`. + +**The friction.** Engines cache per-array state keyed on metadata, so every +mutation site must remember to rebind. We fixed three separate bugs of this +class on the branch (stale engine after `with_config`, after +`update_attributes`, and the `_rebind_engine` sprinkling after +`resize`/`append`). Any future mutation site will silently reintroduce the +bug — the pattern invites it. + +**The change.** One canonical rule: arrays are immutable; every metadata or +config change flows through a single `_with_context` chokepoint that +returns a successor array with a rebound engine. `resize` and friends +become thin wrappers over it. + +## 6. Indexing is fused to the chunk/codec machinery + +**The pattern.** `Indexer` subclasses emit per-chunk projections +(`chunk_coords, chunk_selection, out_selection`) consumed directly by +`CodecPipeline.read/write`. There is no neutral, backend-independent +representation of "what the user selected." + +**The friction.** The engine boundary needed one, so we invented `Region` + +the normalize/post-index sandwich. That works, but the contiguous-box +interchange forces the accepted bounding-box amplification for sparse fancy +selections *on every engine including the default*, and hypothesis promptly +found two facade regressions (integer-axis widening, empty block slices) in +code that re-derives what the indexers already knew. A richer interchange +(point lists, per-chunk plans) has nowhere natural to live because the +indexers speak only "chunk projections." + +**The change.** Promote a selection IR as the internal currency: indexers +*produce* it (Region today; later a small union — box, strided box, point +set), and both the codec pipeline and external engines *consume* it. The +facade then translates user selections once, in one direction, instead of +zarr-python decomposing to chunk projections while the engine layer +re-normalizes to boxes. + +## 7. The data path is numpy-centric despite the buffer abstraction + +**The pattern.** `NDBuffer`/`BufferPrototype` exist precisely to abstract +device buffers, but the surrounding code freely calls `np.asarray`, +`np.broadcast_to`, `np.ascontiguousarray` — and the engine spec itself +settled on "results implement `__array__`". + +**The friction.** The single Critical finding of the final review: the new +facade broke the GPU data path in six places, then twice more in residual +sweeps (`np.generic` scalars, `ascontiguousarray`'s C-level coercion). Each +fix was "keep the operation in the result's own array namespace" — done by +hand, site by site, guarded by a stand-in test because CI has no GPU. + +**The change.** Make the buffer story the lingua franca of the data path: +either extend `BufferPrototype`/`NDBuffer` with the operations the facade +needs (broadcast, patch-assign, astype, scalar extraction) so raw `np.*` +never appears there, or adopt the array-API namespace +(`array_api_compat`) as the internal convention. Plus: keep the +raising-`__array__` stand-in test as a permanent CPU-run guard (done on +this branch — worth upstreaming as a pattern). + +## 8. Two competing accelerator hooks + +**The pattern.** The historical acceleration point is codec-pipeline +injection (`config codec_pipeline.path`, the zarrs-python approach). The +engine layer now coexists with it at a different altitude. + +**The friction.** Conceptual, not mechanical, but real: the codec-pipeline +hook cannot express what engines exist for (native store I/O, sharding-aware +subset reads, sync-native paths), while the engine hook subsumes the +pipeline hook's use case. Documentation and user mental models now carry +both. + +**The change.** Declare engines the public accelerator boundary. Reframe +codec-pipeline injection as an implementation detail of the *default +engine* (or deprecate it once zarrs-python ships an engine), so there is +one story: "an engine owns an array's data path." + +## 9. Pervasive v2/v3 branching + +**The pattern.** `zarr_format == 2` branches at 19 sites in `array.py` +alone (dtype vs `data_type`, order-from-metadata vs order-from-config, +key encoding). + +**The friction.** Every one of those branches had to be faithfully cloned +into the default engine (a review caught the `read_missing_chunks` block +that wasn't), and every future engine author confronts them again. The +zarrista engine adds its own "v3 only" gate on top. + +**The change.** Normalize at the metadata boundary: uniform accessors on +`ArrayMetadata` (`.native_dtype`, `.effective_order(config)`, ...) so the +data path — and engine implementers — never branch on `zarr_format`. + +## 10. Dataclass ergonomics for stateful core classes + +Smaller, but it taxed every task that touched `array.py`: frozen-style +dataclasses with 10 `object.__setattr__` calls, `field(init=False)` slots, +and the `engine`/`engine_spec` name collision. These classes have +constructor logic, lazy state, and invariants — a plain class with an +explicit `__init__` (or attrs) would say what it means. + +## Priority if zarr-python adopts these + +1. §1 + §4 together (single class parameterized by engine, bound to one + context object) — they eliminate the two largest workaround clusters on + the branch (lazy nullable engines, rebinding bugs). +2. §3 (store native-handle protocol) — unlocks third-party stores for + native engines and deletes the isinstance translation layer. +3. §2 (creation-path consolidation) — makes every future cross-cutting + parameter a one-file change. +4. §6 + §7 — quality-of-implementation: kill the double normalization and + the hand-audited GPU path. +5. §8, §9, §10 — cleanups that reduce ongoing tax. + +## Companion asks on the Zarrista side (tracked elsewhere) + +For completeness, the mirror list lives in the design spec: multi-chunk +`store_array_subset` (confirmed easy), a Python store bridge (would relax +§3 from "required" to "optimization"), step>1 selections, v2 reads, and +config hooks so a zarrista engine could honor `read_missing_chunks` instead +of rejecting it. From b5f9cd6e3f349bac753f236f3c1f08f2fa7827ef Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 15:07:04 +0200 Subject: [PATCH 74/75] docs: report on zarrista changes needed for a fluent binding layer Assisted-by: ClaudeCode:claude-fable-5 --- ...23-zarrista-changes-for-fluent-bindings.md | 211 ++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md diff --git a/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md b/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md new file mode 100644 index 0000000000..0f3a5dbbe2 --- /dev/null +++ b/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md @@ -0,0 +1,211 @@ +# What Zarrista should change for a fluent zarr-python binding + +Date: 2026-07-23 +Status: discussion draft (audience: the Zarrista team) +Context: companion to `2026-07-23-zarr-python-blockers-for-engine-integration.md`. +zarr-python now routes array I/O through pluggable engines, and +`zarr.zarrista` implements them over Zarrista's `main` branch (pinned at +`95e47ad`). The binding works and passes a differential suite against the +pure-Python engine, but several Zarrista-side gaps forced Python-side +workarounds, per-read copies, or outright feature rejection. Each item below +states current behavior, the concrete cost it imposed on the binding, and +the ask — ranked by impact. + +## What already works well (please don't change it) + +- **`zarr_metadata` as the metadata currency.** `from_metadata` accepting + the `ArrayMetadataV3` TypedDict means zarr-python's + `ArrayV3Metadata.to_dict()` output is directly consumable — no + serialization shim, no divergent metadata model. This was the single + biggest reason the binding is thin. +- **Native sync and async classes.** zarr-python's sync `Array` now calls + Zarrista's sync API with no event loop anywhere in the caller thread — + only possible because the sync side is genuinely sync. +- **Ndim-preserving `Selection` semantics.** Matches the binding's `Region` + interchange exactly; `read_selection` is a single + `retrieve_array_subset` call with zero dispatch. +- **`from_metadata` constructing without writing.** Lets the binding open + arrays zarr-python already manages without touching the store. + +## 1. Multi-chunk writes: expose `store_array_subset` + +**Today.** Writes are chunk-at-a-time (`store_chunk`). The binding's +`write_selection` therefore decomposes the region over the chunk grid in +Python: full chunks encode directly, partial chunks read-modify-write via +`retrieve_chunk` → numpy patch → `store_chunk`. + +**Cost.** The decomposition arithmetic (overlap slices, edge-chunk +clipping, the "full chunk" test that must also compare against the nominal +chunk shape) was the highest-risk code in the binding — one review +correction and a 65-region differential fuzz were needed to trust it. All +of it is a Python re-implementation of what zarrs already does natively, +serially, with a Python round-trip per chunk. + +**Ask.** Expose zarrs's `store_array_subset` on `Array`/`AsyncArray`. The +binding's write path collapses to one call, RMW moves into Rust with +proper parallelism, and the edge-case surface disappears. (Confirmed easy +in earlier review — this is the formal request.) + +## 2. Unify the sync and async store worlds + +**Today.** Sync `Array` accepts `FilesystemStore | MemoryStore`; async +`AsyncArray` accepts obstore `ObjectStore | icechunk Session`. The sets are +disjoint. + +**Cost.** The disjoint sets leak all the way up zarr-python's stack. A +zarr-python array always carries both a sync and an async face; because no +single store satisfies both Zarrista sides, the binding cannot resolve both +engines eagerly and had to make engine construction lazy on both sides +(deferred store translation, nullable cached engines). Sync users get no +object storage; async users get local files only through obstore's +`LocalStore`. + +**Ask.** Accept the same store set on both sides — most simply, blocking +sync wrappers over the async stores (sync `Array.open` taking an obstore +`ObjectStore`), and/or `FilesystemStore` on the async side. One store +handle usable from both `Array` and `AsyncArray` would let the binding +translate once and bind both engines eagerly. + +## 3. A Python store bridge (callback store) + +**Today.** Only Zarrista-native stores work. Arbitrary zarr-python `Store` +implementations — `MemoryStore`, fsspec-backed stores, test doubles, +anything third-party — cannot back a Zarrista array at all; the binding +fails loudly with "unsupported store." + +**Cost.** `engine="zarrista"` silently partitions the store ecosystem. +`MemoryStore` needs a bespoke error explaining Rust-vs-Python address +spaces; every in-memory test of the binding must use `LocalStore` + +tmp-dirs instead. + +**Ask.** A `PyObjectStore`-style adapter: a Zarrista store that calls back +into a user-supplied Python object implementing get/set/delete/list (zarrs +supports custom stores; our deleted homemade bindings had exactly this +shim). Slower than native paths, but it converts "unsupported store" into +"works, at Python-callback speed" — and makes the whole binding testable +in memory. + +## 4. Writable (or ownership-transferring) read buffers + +**Today.** `Tensor`'s buffer-protocol export is read-only; +`np.asarray(tensor)` yields `writeable=False`. + +**Cost.** zarr-python's API contract is that reads return writable arrays. +The binding now detects read-only numpy results on the fast path and +copies — so every full-box zarrista read pays a copy that the zero-copy +design was meant to avoid. (Latent-bug history: before the guard, `arr[:]` +via zarrista returned read-only arrays and in-place user code broke.) + +**Ask.** Either an ownership-transferring export (`Tensor.into_numpy()` — +Rust relinquishes the allocation, numpy gets a writable array, no copy) or +a documented `to_numpy(writable=True)` that copies inside Rust. The former +restores true zero-copy for the common read path; the latter at least +moves the copy where it's visible and optimizable. + +## 5. A missing-chunk policy on reads + +**Today.** `retrieve_chunk`/`retrieve_array_subset` silently materialize +fill values for absent chunks. There is no way to ask "and tell me (or +raise) if a chunk was missing." + +**Cost.** zarr-python has `read_missing_chunks=False` (raise +`ChunkNotFoundError` on uninitialized chunks). The default engine honors +it; the zarrista engine cannot, so the binding *rejects* that config with +`UnsupportedEngineError` — a documented capability gap between engines. + +**Ask.** A codec-option (or retrieve variant) with a missing-chunk policy: +`fill` (today's behavior) | `error`, or alternatively return a +chunks-present mask alongside the data. Either form lets the binding honor +the config instead of refusing it. + +## 6. Make `VariableArray → numpy` either work or raise + +**Today.** vlen results (`VariableArray`) export Arrow capsules; the +planned numpy export (copying) has not landed. Critically, +`np.asarray(variable_array)` today neither works nor raises — it silently +produces a wrong-shape 0-d object array. + +**Cost.** The binding had to type-gate every decode (`Tensor`-only) because +the generic numpy coercion path silently corrupts vlen reads — this was +caught by a review probe, not by an exception. + +**Ask.** Two parts: (a) land the numpy export for `VariableArray` (a copy +is fine to start) so vlen dtypes work through the binding; (b) until then, +make `__array__` on non-exportable types raise `TypeError` rather than +silently misconverting — silent wrong answers are the worst failure mode a +binding can inherit. + +## 7. Strided selections (`step > 1`) + +**Today.** `Selection` supports integers and step-1 slices only. + +**Cost.** Any strided read (`arr[::2]`) leaves the fast path: the binding +fetches the step-1 bounding box and post-indexes in Python — transferring +and decoding up to `step×` more data than needed. + +**Ask.** Accept arbitrary positive steps in `Selection` (zarrs subsets can +express this via per-chunk mapping even if the core type is contiguous). +Negative steps stay a Python-side concern; positive strides are the common +scientific-access pattern worth doing in Rust. + +## 8. Subchunk-level writes for sharded arrays + +**Today.** Reads are shard-aware (`retrieve_subchunk` reads one inner +chunk without touching the rest of the shard), but there is no +`store_subchunk` — writes operate on whole chunks, i.e. whole shards. + +**Cost.** A one-element write into a large shard round-trips the entire +shard through Python (`retrieve_chunk` of the shard → patch → +`store_chunk`). The read side's nice granularity has no write mirror. + +**Ask.** `store_subchunk` (or make `store_array_subset` from §1 +shard-aware, which subsumes this). Sharded-append workloads are where +zarrs's performance story is strongest; the write path should match. + +## 9. Zarr v2 read support + +**Today.** `from_metadata` accepts `ArrayMetadataV3` only. + +**Cost.** The binding hard-rejects v2 arrays (`UnsupportedEngineError`), +so mixed-version hierarchies silently split into "accelerated" and +"not accelerated" populations. + +**Ask.** zarrs supports a V3-compatible subset of V2; exposing +`from_metadata` for the v2 document (or an internal v2→v3-view conversion) +would let the binding serve most real-world v2 data read-only, which is +the dominant v2 use case. + +## 10. Smaller ergonomic papercuts + +- **`ArrayBytes` rejects multi-dimensional buffers.** The binding flattens + every chunk with `reshape(-1).view(np.uint8)` before `store_chunk`. + Accepting any C-contiguous buffer (validating length against the chunk + shape) would remove a copy-risk and a line of ceremony per write. +- **Path convention.** Zarrista paths are `/`-prefixed with `/` as root; + zarr-python paths are `""`-rooted relatives. The binding normalizes with + `"/" + path.strip("/")` everywhere — accepting both forms would delete a + papercut every consumer rediscovers. +- **Icechunk sessions are reconstructed, not shared.** The session is + serialized into the Rust extension and runs as a separate icechunk + instance; `in_memory_storage()` cannot work, and writes made through the + reconstructed session interact opaquely with the Python-side session's + transaction state. A shared-instance handoff (or a documented + commit-visibility contract) would make icechunk + zarrista trustworthy + for read-write use. +- **Release cadence.** The binding pins a git commit. A beta release with + §1 (and ideally §4/§6) would let zarr-python depend on + `zarrista>=0.1.0bN` instead of a SHA. + +## Priority from the binding's perspective + +1. §1 `store_array_subset` — deletes the riskiest Python code in the + binding and is already agreed to be easy. +2. §4 writable/ownership read buffers — removes a copy from every fast-path + read; the zero-copy story is currently negated by the writability gap. +3. §2 + §3 store-world unification and the Python bridge — these two + dissolve the store-translation layer and most of the lazy-engine + machinery on the zarr-python side. +4. §5 missing-chunk policy and §6 vlen export — close the two documented + capability gaps between the zarrista and default engines. +5. §7–§9 — performance breadth (strided, sharded writes, v2). +6. §10 — papercuts, any time. From 3ff999dabc008c482a6a9e19fefb42c6aaafc9d0 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Thu, 23 Jul 2026 15:21:23 +0200 Subject: [PATCH 75/75] docs: ground the writable-buffer ask in DLPack semantics, not a new protocol Assisted-by: ClaudeCode:claude-fable-5 --- ...23-zarrista-changes-for-fluent-bindings.md | 31 ++++++++++++++++--- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md b/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md index 0f3a5dbbe2..5caadda8a5 100644 --- a/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md +++ b/docs/superpowers/specs/2026-07-23-zarrista-changes-for-fluent-bindings.md @@ -96,11 +96,32 @@ copies — so every full-box zarrista read pays a copy that the zero-copy design was meant to avoid. (Latent-bug history: before the guard, `arr[:]` via zarrista returned read-only arrays and in-place user code broke.) -**Ask.** Either an ownership-transferring export (`Tensor.into_numpy()` — -Rust relinquishes the allocation, numpy gets a writable array, no copy) or -a documented `to_numpy(writable=True)` that copies inside Rust. The former -restores true zero-copy for the common read path; the latter at least -moves the copy where it's visible and optimizable. +**Ask.** Use DLPack's existing semantics — no new protocol. Of the three +standard interchange models, only one fits writable hand-off: + +- The **buffer protocol** is shared-ownership-by-refcount; writability is a + per-export decision (`PyBUF_WRITABLE`). A writable export is sound iff + the Rust side never touches the allocation after exporting — acceptable + as a fallback, but the safety condition is invisible convention. +- **Arrow C data** (already used for `VariableArray`) is release-callback + ownership over **immutable-by-contract** buffers; it has no writable + story and should not be bent into one. +- **DLPack** is built for this. `Tensor.__dlpack__` already exists and + speaks the versioned protocol (`max_version`). Two spec-compliant + options, both with heavy precedent: + 1. *Shared writable view (the PyTorch/CuPy pattern)*: export the + `DLManagedTensorVersioned` **without** `DLPACK_FLAG_BITMASK_READ_ONLY`, + with a deleter that holds a strong reference to the `Tensor`. + `np.from_dlpack` then yields a writable ndarray aliasing the Rust + allocation; sound because a freshly decoded result is never touched + by Rust again. Smallest change. + 2. *True move*: the deleter owns the allocation directly (drops the + `Vec`); the `Tensor` tombstones itself and further access raises. The + capsule's one-shot convention (`used_dltensor` rename) already + enforces single consumption on the consumer side. + +Either way, the binding's read path becomes `np.from_dlpack(result)` on +the fast path and the writability copy-guard never fires for zarrista. ## 5. A missing-chunk policy on reads