From d916f759dca7ee41bfa56f51a2295b8b0e90a81a Mon Sep 17 00:00:00 2001 From: Ilan Gold Date: Tue, 14 Jul 2026 17:13:03 +0200 Subject: [PATCH 01/16] chore: add Ilan Gold to the team list (#4145) --- TEAM.md | 1 + 1 file changed, 1 insertion(+) diff --git a/TEAM.md b/TEAM.md index dc22a1ee87..ce9de1d486 100644 --- a/TEAM.md +++ b/TEAM.md @@ -11,6 +11,7 @@ - @dcherian (Deepak Cherian) - @TomAugspurger (Tom Augspurger) - @maxrjones (Max Jones) +- @ilan-gold (Ilan Gold) ## Emeritus core-developers - @alimanfoo (Alistair Miles) From 3c43e3e980fa10ae73b51a662cffba820c35467d Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 15 Jul 2026 09:56:50 +0200 Subject: [PATCH 02/16] docs: add roadmap page outlining future plans Adds a Roadmap page to the documentation describing the goals for the next major cycle of work ("v4"), the intended changes per theme, and the three-stream release model (additive minors, deprecations, one minimal removals-only major). Assisted-by: ClaudeCode:claude-fable-5 --- changes/+roadmap.doc.md | 1 + docs/roadmap.md | 324 ++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 326 insertions(+) create mode 100644 changes/+roadmap.doc.md create mode 100644 docs/roadmap.md diff --git a/changes/+roadmap.doc.md b/changes/+roadmap.doc.md new file mode 100644 index 0000000000..8a473acac9 --- /dev/null +++ b/changes/+roadmap.doc.md @@ -0,0 +1 @@ +Added a Roadmap page to the documentation outlining future plans and intended changes to the library. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000000..b68c86ad5e --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,324 @@ +# Roadmap + +This page describes where Zarr-Python is headed: the goals for the next major +cycle of work, the changes we intend to make, and how those changes will be +released. It is a living document; discussion and counter-proposals are welcome +on the +[zarr-python issue tracker](https://github.com/zarr-developers/zarr-python/issues). + +*The history of this roadmap, including the detailed technical proposals it +was distilled from, can be traced in the +[zarr-python-planning](https://github.com/d-v-b/zarr-python-planning) +repository.* + +!!! note + + This roadmap reflects the current thinking of the core developers. It is a + statement of direction, not a schedule: the work ships when it is ready, + and individual items may change shape as the proposals are discussed and + refined. + +## Where we are + +The [3.0 release](https://github.com/zarr-developers/zarr-python/releases/tag/v3.0.0) +was a total redesign of the library's internals, with three goals: full support +for the Zarr V2 and V3 storage formats, storage APIs ergonomic for high-latency +(cloud) storage, and backwards compatibility with Zarr-Python 2.x where +possible. Those goals were largely achieved, and more than a year on, the +2.x → 3.x transition is effectively resolved. + +The 3.x redesign was carried out under hard backwards-compatibility +constraints, and it inherited many structural patterns from the 2.x +implementation it replaced. The library has never had a release cycle whose +primary goal was the *shape* of the internals. The next body of work — which we +call **"v4"** — is that overdue investment. + +## Goals + +If the 3.0 goals could be sloganized as "migrate to Zarr V3, and improve cloud +storage support", the slogan for the v4 goals is: +**"support a Zarr-based Python ecosystem for chunked arrays"**. Zarr-Python +should be *foundational* for the growing number of Python packages that work +with data in the Zarr format. Concretely, that means pushing in these +directions: + +- Give Zarr-Python users excellent performance, out of the box. +- Make Zarr-Python APIs ergonomic and useful for developers. +- Expand our scope to cover vital quality-of-life routines like data copying, + rechunking, and the like. +- Support the growth of Python tools across all levels of the Zarr stack. +- Accelerate the implementation of new codecs, chunk grids, chunk key + encodings, etc. + +An important design input: [zarrs](https://github.com/zarrs/zarrs) (Rust) and +[TensorStore](https://github.com/google/tensorstore) (C++) are two independent +Zarr implementations that have converged on the same architectural patterns — +sync-first codec APIs, per-codec concurrency budgets, adaptive sharded-read +strategies, request deduplication, conditional reads. We treat them as +complementary rather than competitive: Zarr-Python aims to be the best +pure-Python Zarr implementation *and* the best wrapper around the +compiled-language implementations, so that users who need native throughput can +get it without leaving the Zarr-Python API surface. + +## The Zarr stack + +Different applications need different levels of Zarr support: a convention +validator only needs to read metadata documents; a visualization tool may only +need read-only array access; other tools need everything. We think of this as a +"Zarr stack", from most abstract to most concrete: + +1. **Conventions** — domain-specific schemas built on top of Zarr (OME-NGFF, + GeoZarr, anndata-zarr). +2. **Groups** — Zarr hierarchies, traversal, group-level attributes. +3. **Arrays** — the user-facing array object, plus indexing and slicing. +4. **Chunk decoding** — the codec pipeline. +5. **Chunk addressing** — chunk grids and key encodings that map array + coordinates to store keys. +6. **Stores** — the key-value layer. +7. **Metadata** — pure data documents describing arrays and groups. + +Today, Zarr-Python is a monolith that serves every level: a consumer who only +needs metadata handling has to install the full dependency footprint of the +whole library, and a faster chunk-decoding implementation cannot plug in +without re-implementing the layers above it. The v4 direction is to re-shape +Zarr-Python around the stack, so that each level is something you can depend +on, conform to, or replace, without buying every other level: + +- **A focused package per level** — `zarr-metadata`, `zarr-store`, + `zarr-codec`, `zarr-dtype`, with `zarr` as the facade that composes them. + The first of these, + [`zarr-metadata`](https://pypi.org/project/zarr-metadata/), is already + published. +- **A documented interface per level** — capability protocols for stores, a + small stateless codec API, pure-data dtypes. +- **A conformance suite per level** — so that alternative implementations of a + level can verify they behave correctly. +- **Engine pluggability at the chunk-decoding level** — alternative engines + (zarrs, TensorStore) can take over IO without re-implementing hierarchy + traversal, indexing, or metadata handling. + +## What we intend to change + +Each theme below is backed by a detailed technical proposal; the summaries +here describe the intended end state. + +### Foundation: a functional core + +Refactor the internals around a *functional core* — pure data structures and +pure functions for the algebra of Zarr (metadata, chunk layouts, slice +planning, codec walking) — with the side-effecting protocols (stores, codecs) +at the edges. This is an internal change that makes the per-level package split +implementable and provides a clean substrate for engine pluggability. + +### Foundation: a formal hierarchy layer + +Name and specify the layer that sits between the store API (key-agnostic +bytes) and the user-facing `Array` / `Group` facade, as a small set of typed +verbs (`read_array_metadata`, `write_chunk`, `list_children`, +`read_selection`, ...). Alternative engines implement the verbs end-to-end; +hierarchy-aware caching wraps them; chunk-introspection APIs expose them. + +### Codecs + +The current codec API wraps every codec in an unnecessary async layer (a +profiling hotspot), bakes batching into every signature, and forces output +allocation even when the caller has a buffer ready. Rewrite the codec API as a +small, stateless capability bundle — sync-first encode/decode, single-element +signatures, optional `decode_into`, capability flags — decoupled from the rest +of the library, with a compatibility shim for existing codecs and clear paths +for migrating Zarr V2 codecs that still have no V3 equivalent. + +### Stores + +The store abstraction conflates lifecycle, path handling, sync/async, +capability advertisement, and read-only semantics into one inheritance +hierarchy, and the resulting friction has produced a recurring stream of +regressions. Redesign stores as composable capability protocols (`Get`, `Put`, +`List`, ...) with composable wrappers (caching, range coalescing, retries), +transactional semantics, and a shared conformance suite that backends and +wrappers parameterize. + +### Performance + +A cross-cutting theme that ties the codec, store, and functional-core work +into one performance story: typed, library-owned concurrency resources with +dask-safe defaults; synchronous codec encode/decode on the default read path; +range coalescing; pre-allocated decode buffers; in-flight request +deduplication; ETag-style conditional reads; a unified caching substrate with +sensible defaults; an adaptive whole-shard-vs-coalesced read strategy; and +pluggable high-performance backends (zarrs, TensorStore) selectable with a +keyword argument, so the same `Array` and `Group` — and the same Xarray, Dask, +and napari integrations — work at native throughput. A benchmark suite for the +target access patterns lands first, so every performance lever ships with +before/after numbers. + +### Lazy indexing + +`Array.__getitem__` performs IO eagerly and returns NumPy, which makes Zarr +arrays the odd one out among modern array libraries and blocks participation in +the [Python Array API](https://data-apis.org/array-api/) ecosystem. Add an +opt-in `array.lazy[...]` accessor backed by a stable coordinate-mapping algebra +(the `IndexTransform` work in +[#3906](https://github.com/zarr-developers/zarr-python/pull/3906)), plus a +small query planner that turns chained selections into a single IO plan before +any chunks are fetched. No new array type is introduced. Whether the *default* +of bare `array[...]` ever flips from eager to lazy is an explicit, separate +decision — see [decision points](#decision-points) below. + +### Data types + +First-class support for ML-specific dtypes — `bfloat16`, the `float8` +variants, packed `int4`/`uint4` — via +[`ml_dtypes`](https://github.com/jax-ml/ml_dtypes), using the exact identifiers +registered in `zarr-extensions` so the data stays readable by other +implementations. Ragged arrays, variable-length strings, and an investigation +of Apache Arrow as a substrate for the dtypes the Array API cannot express are +follow-on work on the same substrate. + +### Device-agnostic IO + +Make Zarr-Python's IO surfaces device-agnostic rather than adding GPU support +as a bolted-on feature: stores and codecs grow APIs for writing into a +caller-provided buffer (`read_into`, `decode_into`), and the `Array` facade +returns array-like objects in the user's chosen Array API namespace. GPU +support falls out once the assumption of CPU destinations is removed, and CPU +paths get faster too, because pre-allocated output buffers eliminate per-chunk +allocation. + +### Observability + +Two pillars: **performance metrics and tracing** (a small library-owned +`Metrics` object plus OpenTelemetry auto-instrumentation across stores, codecs, +caches, and the engine boundary) and **stored-state introspection** (public +APIs for asking about chunk-level structure, materialization, byte ranges, and +storage footprint without reading the chunks — the surface projects like +VirtualiZarr and Kerchunk have been asking for). + +### Configuration, registries, and plugins + +Move configuration from "global mutable state read implicitly" to "typed data +passed explicitly": a typed config object replacing the untyped global `donfig` +dict, array-scoped runtime config passed at open time, a registry redesign that +addresses implementations by stable identity and resolves plugin name-conflicts +deliberately, and named profiles replacing global mutators. This substrate is +where the performance-lever defaults (concurrency, caching, engine selection) +will live, so it lands early. + +### Consolidated metadata + +Consolidated metadata is essential for performance on high-latency storage and +widely used downstream, but the current support has open design questions +around codec/dtype/grid representations, write-time invalidation, and V2/V3 +migration. A stored V3 representation is a *format* decision, so the design +pass routes through the Zarr spec process (ZEP), co-designed with Xarray. + +### Coordinated and distributed writes + +Give the two patterns that actually produce large Zarr archives — parallel +disjoint-region writes and append-along-axis growth — a design home: disjoint +chunk-aligned region writes with alignment *checked* rather than assumed, a +create-then-hand-out-regions primitive, and single-writer resize/append, all on +plain Zarr V3. Stronger guarantees (atomicity, reader isolation, concurrent +appenders) are enabled through the seam a transactional engine such as +[Icechunk](https://icechunk.io/) builds on, rather than implemented in +Zarr-Python itself. + +### Missing APIs + +User-facing conveniences that users have been asking for, in some cases for +years: hierarchy navigation helpers, chunk introspection, explicit constructors +replacing `mode=`, a typed exception hierarchy, rich reprs, context-manager +support, data copying, and an in-library rechunking primitive. + +## How the work will be released + +**"v4" names this whole body of work, delivered across many releases — it is +not a single "4.0" feature release.** The work is organized into three streams +that run in parallel: + +| Stream | Release vehicle | Scope | +|---|---|---| +| **Additive value** | 3.x minor releases, shipping continuously | The overwhelming majority of the plan, including the entire foundation. No migration required. | +| **Deprecation accumulation** | Warnings across the 3.x line | Each surface is deprecated only *after* its additive replacement has shipped, so users always have a migration target before they see a warning. | +| **Breaking removals** | One minimal, late major release (4.0.0) | Removal of the deprecated surfaces, and *only* those, after deprecation windows have elapsed and downstream libraries have had release windows to adapt. | + +The additive stream is itself roughly ordered: + +1. **Ship-now wins** — dependency-free improvements that land first: the + benchmark suite, store-layer range coalescing, in-flight request + deduplication, the sync codec path on default reads, ML dtype support, + constructor and display UX. +2. **Foundation** — the functional-core refactor, the per-level package split, + the new stores API, the hierarchy verbs, the typed configuration substrate, + the full concurrency and caching rework, and the codec API rewrite. Mostly + invisible to users, all additive. +3. **User-facing surface** — opt-in lazy indexing and the query planner, + device-agnostic IO, observability, chunk introspection, and the zarrs and + TensorStore engine wrappers, built on the foundation. + +The eventual 4.0.0 release contains only removals whose replacements shipped +earlier: the legacy `Store` ABC and the `Buffer`/`prototype` read contract, the +`mode=` constructors, the internal `sync()` bridge, and — conditionally — the +eager `array[...]` path. Nothing new is delivered there; it is the only release +downstream maintainers must treat as breaking, and it arrives after the value +has already been delivered additively. + +### Backwards-compatibility commitments + +The v4 work changes the public API: methods will be renamed, signatures will +change, deprecated patterns will be removed, and the codec and store APIs will +be rewritten. We believe the changes are worth the cost, and we commit to the +following: + +- **Conformance with community standards.** Where a relevant cross-language + standard exists, we conform to it: the Python Array API at the array surface, + the Zarr V3 spec and its extensions at the storage layer, OpenTelemetry for + tracing, and standard buffer-protocol and device-interop conventions for + device-agnostic IO. +- **Functional coverage.** Anything you can do in Zarr-Python 3.x you will + still be able to do once the v4 work has landed — sometimes through a renamed + API, but the capability is preserved. We will not remove the ability to read + or write any Zarr-format data that 3.x supports. +- **A deprecation window for every change.** Renames and removals land through + deprecation cycles, and downstream libraries (Xarray, Dask, napari) get + release windows to absorb each change before the next one lands. + +### Decision points + +Flipping the default of bare `array[...]` from eager to lazy is the single +highest-migration-cost item in the plan, so it is handled as an explicit +decision, not bundled into the additive work. The opt-in `array.lazy[...]` +accessor ships first, with no default change. Whether the default ever flips +hinges on whether Array API conformance at the bare-`__getitem__` surface turns +out to be a hard requirement; if it does, the flip happens as a long-window +deprecation with an explicit eager escape hatch and downstream coordination — +never as a reason to adopt a major version. + +### Out of scope + +- **Persisted hierarchy links** (HDF5-style soft/hard/external links) — would + require defining a new on-disk format unilaterally; needs a Zarr Enhancement + Proposal first. +- **Declarative hierarchy schema validation** as a shipped feature — likely a + separate package layered on `zarr-metadata`, deferred. +- **Cross-process shared-memory caching** — the caching substrate is designed + not to foreclose it, but it does not ship now. +- **Further V2→V3 migration tooling** — the 2.x → 3.x transition is + effectively resolved. + +## How to get involved + +- **Discuss the plans.** Comments and counter-proposals on any of the themes + above are welcome on the + [issue tracker](https://github.com/zarr-developers/zarr-python/issues) and in + the [developer chat](https://ossci.zulipchat.com/). +- **Review in-flight work.** The `IndexTransform` algebra that lazy indexing is + built on is in review at + [#3906](https://github.com/zarr-developers/zarr-python/pull/3906). +- **Weigh in as a downstream maintainer.** If your project's use of + Zarr-Python would be affected by the codec API rewrite, the stores rewrite, + or the lazy-indexing work, the planning phase is the time to surface + workloads or patterns that don't fit. +- **Participate in the spec process.** Several open questions (persisted + hierarchy links, ML dtype identifiers, consolidated metadata) ultimately need + [Zarr Enhancement Proposals](https://zarr.dev/zeps/). diff --git a/mkdocs.yml b/mkdocs.yml index 7a4bfa35ef..01d7d96ec7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -72,6 +72,7 @@ nav: - api/zarr/testing/strategies.md - api/zarr/testing/utils.md - release-notes.md + - roadmap.md - contributing.md watch: - src/zarr From 9304a064db03f1a390ed99e9c081219b9bffa0d7 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Wed, 15 Jul 2026 22:03:11 +0200 Subject: [PATCH 03/16] docs: refine roadmap --- docs/roadmap.md | 31 +++++-------------------------- 1 file changed, 5 insertions(+), 26 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index b68c86ad5e..7a0c770b78 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -154,9 +154,9 @@ before/after numbers. ### Lazy indexing -`Array.__getitem__` performs IO eagerly and returns NumPy, which makes Zarr -arrays the odd one out among modern array libraries and blocks participation in -the [Python Array API](https://data-apis.org/array-api/) ecosystem. Add an +`Array.__getitem__` performs IO eagerly and returns NumPy arrays, which makes Zarr +arrays the odd one out among modern array libraries and blocks compliance with +the [Python Array API](https://data-apis.org/array-api/) standard. Add an opt-in `array.lazy[...]` accessor backed by a stable coordinate-mapping algebra (the `IndexTransform` work in [#3906](https://github.com/zarr-developers/zarr-python/pull/3906)), plus a @@ -204,13 +204,6 @@ deliberately, and named profiles replacing global mutators. This substrate is where the performance-lever defaults (concurrency, caching, engine selection) will live, so it lands early. -### Consolidated metadata - -Consolidated metadata is essential for performance on high-latency storage and -widely used downstream, but the current support has open design questions -around codec/dtype/grid representations, write-time invalidation, and V2/V3 -migration. A stored V3 representation is a *format* decision, so the design -pass routes through the Zarr spec process (ZEP), co-designed with Xarray. ### Coordinated and distributed writes @@ -282,6 +275,7 @@ following: - **A deprecation window for every change.** Renames and removals land through deprecation cycles, and downstream libraries (Xarray, Dask, napari) get release windows to absorb each change before the next one lands. +- **Generous legacy support** If necessary, we can keep old code around in a `legacy` module. Pydantic used a similar strategy to manage their 2.0 release: see https://pydantic.dev/docs/validation/dev/get-started/migration/#using-pydantic-v1-features-in-a-v1v2-environment. ### Decision points @@ -294,18 +288,6 @@ out to be a hard requirement; if it does, the flip happens as a long-window deprecation with an explicit eager escape hatch and downstream coordination — never as a reason to adopt a major version. -### Out of scope - -- **Persisted hierarchy links** (HDF5-style soft/hard/external links) — would - require defining a new on-disk format unilaterally; needs a Zarr Enhancement - Proposal first. -- **Declarative hierarchy schema validation** as a shipped feature — likely a - separate package layered on `zarr-metadata`, deferred. -- **Cross-process shared-memory caching** — the caching substrate is designed - not to foreclose it, but it does not ship now. -- **Further V2→V3 migration tooling** — the 2.x → 3.x transition is - effectively resolved. - ## How to get involved - **Discuss the plans.** Comments and counter-proposals on any of the themes @@ -318,7 +300,4 @@ never as a reason to adopt a major version. - **Weigh in as a downstream maintainer.** If your project's use of Zarr-Python would be affected by the codec API rewrite, the stores rewrite, or the lazy-indexing work, the planning phase is the time to surface - workloads or patterns that don't fit. -- **Participate in the spec process.** Several open questions (persisted - hierarchy links, ML dtype identifiers, consolidated metadata) ultimately need - [Zarr Enhancement Proposals](https://zarr.dev/zeps/). + workloads or patterns that don't fit. \ No newline at end of file From 91a8ec06dc4d475fe40691e0ab668dd655d3751a Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Thu, 16 Jul 2026 16:34:50 +0200 Subject: [PATCH 04/16] docs: 3.3.0 release notes (#4148) * chore(deps): bump the actions group across 1 directory with 8 updates (#176) Bumps the actions group with 8 updates in the / directory: | Package | From | To | | --- | --- | --- | | [prefix-dev/setup-pixi](https://github.com/prefix-dev/setup-pixi) | `0.9.5` | `0.9.6` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [github/issue-metrics](https://github.com/github/issue-metrics) | `4.2.2` | `4.2.7` | | [j178/prek-action](https://github.com/j178/prek-action) | `2.0.3` | `2.0.4` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `7.0.0` | `7.0.1` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.1` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.0` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.3` | `0.5.6` | Updates `prefix-dev/setup-pixi` from 0.9.5 to 0.9.6 - [Release notes](https://github.com/prefix-dev/setup-pixi/releases) - [Commits](https://github.com/prefix-dev/setup-pixi/compare/1b2de7f3351f171c8b4dfeb558c639cb58ed4ec0...5185adfbffb4bd703da3010310260805d89ebb11) Updates `codecov/codecov-action` from 6.0.0 to 6.0.1 - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codecov/codecov-action/compare/57e3a136b779b570ffcdbf80b3bdc90e7fab3de2...e79a6962e0d4c0c17b229090214935d2e33f8354) Updates `github/issue-metrics` from 4.2.2 to 4.2.7 - [Release notes](https://github.com/github/issue-metrics/releases) - [Commits](https://github.com/github/issue-metrics/compare/c9e9838147fd355dace335ba787f01b6641a400a...1e38d5e62363e14db8019ed7d106b9855bdba6cc) Updates `j178/prek-action` from 2.0.3 to 2.0.4 - [Release notes](https://github.com/j178/prek-action/releases) - [Commits](https://github.com/j178/prek-action/compare/6ad80277337ad479fe43bd70701c3f7f8aa74db3...bdca6f102f98e2b4c7029491a53dfd366469e33d) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v7...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) Updates `actions/download-artifact` from 7.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v7...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) Updates `pypa/gh-action-pypi-publish` from 1.13.0 to 1.14.0 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/v1.13.0...cef221092ed1bacb1cc03d23a2d87d1d172e277b) Updates `zizmorcore/zizmor-action` from 0.5.3 to 0.5.6 - [Release notes](https://github.com/zizmorcore/zizmor-action/releases) - [Commits](https://github.com/zizmorcore/zizmor-action/compare/b1d7e1fb5de872772f31590499237e7cce841e8e...5f14fd08f7cf1cb1609c1e344975f152c7ee938d) --- updated-dependencies: - dependency-name: prefix-dev/setup-pixi dependency-version: 0.9.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github/issue-metrics dependency-version: 4.2.7 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: j178/prek-action dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: zizmorcore/zizmor-action dependency-version: 0.5.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * docs: 3.3.0 release notes Add missing changelog fragments for #3955 (datetime64/timedelta64 V3 metadata types), #3966 (writes to 0-dimensional sharded arrays), and the public alias renames that accompanied the #3963/#3968 enum deprecations, then build the 3.3.0 release notes with towncrier, consuming all fragments accumulated since v3.2.1. Co-Authored-By: Claude Fable 5 --------- Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Claude Fable 5 --- changes/202.bugfix.md | 1 - changes/214.misc.md | 1 - changes/215.misc.md | 1 - changes/2929.bugfix.md | 3 - changes/3004.feature.md | 4 - changes/3009.doc.md | 1 - changes/3417.bugfix.md | 4 - changes/3469.bugfix.md | 1 - changes/3826.feature.md | 1 - changes/3885.bugfix.md | 1 - changes/3885.feature.1.md | 1 - changes/3885.feature.md | 1 - changes/3899.bugfix.md | 7 -- changes/3908.misc.md | 1 - changes/3925.feature.md | 1 - changes/3954.bugfix.md | 1 - changes/3963.removal.md | 6 -- changes/3968.removal.md | 17 ----- changes/3972.misc.md | 1 - changes/3973.removal.md | 1 - changes/3975.misc.md | 1 - changes/3977.bugfix.md | 1 - changes/3979.misc.md | 1 - changes/3984.doc.md | 4 - changes/3987.feature.md | 1 - changes/3990.misc.md | 5 -- changes/3998.misc.md | 1 - changes/4000.misc.md | 1 - changes/4001.misc.md | 6 -- changes/4003.bugfix.md | 19 ----- changes/4016.bugfix.md | 1 - changes/4032.bugfix.md | 1 - changes/4046.misc.md | 1 - changes/4052.doc.md | 21 ----- changes/4053.doc.md | 1 - changes/4054.misc.md | 1 - changes/4059.bugfix.md | 1 - changes/4073.misc.md | 1 - changes/4074.bugfix.md | 7 -- changes/4086.misc.md | 1 - changes/4116.bugfix.md | 1 - changes/4132.doc.md | 6 -- changes/4133.doc.md | 1 - changes/4138.misc.md | 1 - changes/4141.bugfix.md | 1 - docs/release-notes.md | 156 ++++++++++++++++++++++++++++++++++++++ 46 files changed, 156 insertions(+), 141 deletions(-) delete mode 100644 changes/202.bugfix.md delete mode 100644 changes/214.misc.md delete mode 100644 changes/215.misc.md delete mode 100644 changes/2929.bugfix.md delete mode 100644 changes/3004.feature.md delete mode 100644 changes/3009.doc.md delete mode 100644 changes/3417.bugfix.md delete mode 100644 changes/3469.bugfix.md delete mode 100644 changes/3826.feature.md delete mode 100644 changes/3885.bugfix.md delete mode 100644 changes/3885.feature.1.md delete mode 100644 changes/3885.feature.md delete mode 100644 changes/3899.bugfix.md delete mode 100644 changes/3908.misc.md delete mode 100644 changes/3925.feature.md delete mode 100644 changes/3954.bugfix.md delete mode 100644 changes/3963.removal.md delete mode 100644 changes/3968.removal.md delete mode 100644 changes/3972.misc.md delete mode 100644 changes/3973.removal.md delete mode 100644 changes/3975.misc.md delete mode 100644 changes/3977.bugfix.md delete mode 100644 changes/3979.misc.md delete mode 100644 changes/3984.doc.md delete mode 100644 changes/3987.feature.md delete mode 100644 changes/3990.misc.md delete mode 100644 changes/3998.misc.md delete mode 100644 changes/4000.misc.md delete mode 100644 changes/4001.misc.md delete mode 100644 changes/4003.bugfix.md delete mode 100644 changes/4016.bugfix.md delete mode 100644 changes/4032.bugfix.md delete mode 100644 changes/4046.misc.md delete mode 100644 changes/4052.doc.md delete mode 100644 changes/4053.doc.md delete mode 100644 changes/4054.misc.md delete mode 100644 changes/4059.bugfix.md delete mode 100644 changes/4073.misc.md delete mode 100644 changes/4074.bugfix.md delete mode 100644 changes/4086.misc.md delete mode 100644 changes/4116.bugfix.md delete mode 100644 changes/4132.doc.md delete mode 100644 changes/4133.doc.md delete mode 100644 changes/4138.misc.md delete mode 100644 changes/4141.bugfix.md diff --git a/changes/202.bugfix.md b/changes/202.bugfix.md deleted file mode 100644 index 9c9bd40f21..0000000000 --- a/changes/202.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Stop emitting an `UnstableSpecificationWarning` when serializing the `struct` data type to Zarr V3 metadata. The `struct` data type now has a stable Zarr V3 specification. The legacy `structured` alias and the unspecified `null_terminated_bytes`, `raw_bytes`, and `variable_length_bytes` data types continue to warn. diff --git a/changes/214.misc.md b/changes/214.misc.md deleted file mode 100644 index 8611362e84..0000000000 --- a/changes/214.misc.md +++ /dev/null @@ -1 +0,0 @@ -Bind the shared moto S3 test server to an ephemeral port instead of a fixed one. The fixed port deadlocked `pytest -n auto`: session-scoped fixtures run once per xdist worker, so concurrent workers raced to bind the same port and the losers blocked forever in `ThreadedMotoServer.start()`. diff --git a/changes/215.misc.md b/changes/215.misc.md deleted file mode 100644 index f8df7cc632..0000000000 --- a/changes/215.misc.md +++ /dev/null @@ -1 +0,0 @@ -Enable pytest's `strict = true` config option (strict config, markers, xfail, and parametrization ids), replacing the `--strict-config`/`--strict-markers` addopts flags that pytest silently ignored before 9.1, and fix the seven duplicate parametrization ids it surfaced — including restoring float-JSON roundtrip cases that were meant to cover numpy scalars but ran plain-float cases twice instead. Also arm pytest's faulthandler watchdog (`faulthandler_timeout = 600` with `faulthandler_exit_on_timeout`) so a deadlocked test dumps every thread's traceback and fails the run instead of hanging indefinitely. diff --git a/changes/2929.bugfix.md b/changes/2929.bugfix.md deleted file mode 100644 index 533a5f86c1..0000000000 --- a/changes/2929.bugfix.md +++ /dev/null @@ -1,3 +0,0 @@ -Fix equality comparison of `ArrayV2Metadata` and `ArrayV3Metadata` objects with a -`NaN` fill value. Such objects are now compared by their JSON-serialized form, so two -otherwise-identical metadata objects with a `NaN` (or infinite) fill value compare equal. diff --git a/changes/3004.feature.md b/changes/3004.feature.md deleted file mode 100644 index 9d3816a50c..0000000000 --- a/changes/3004.feature.md +++ /dev/null @@ -1,4 +0,0 @@ -Optimizes reading multiple chunks from a shard. Serial calls to `Store.get()` -in the sharding codec have been replaced with a single call to -`Store.get_ranges()`, which coalesces nearby byte ranges and fetches them -concurrently. diff --git a/changes/3009.doc.md b/changes/3009.doc.md deleted file mode 100644 index 777672b77b..0000000000 --- a/changes/3009.doc.md +++ /dev/null @@ -1 +0,0 @@ -Document the changes to `zarr.errors` in the 3.0 migration guide, including the removal of v2 exception classes and the introduction of `NodeNotFoundError`. diff --git a/changes/3417.bugfix.md b/changes/3417.bugfix.md deleted file mode 100644 index be5b44f3e9..0000000000 --- a/changes/3417.bugfix.md +++ /dev/null @@ -1,4 +0,0 @@ -Fixed `BytesCodec.from_dict` so that `BytesCodec` instances roundtrip to / from -their dict representation. `BytesCodec.from_dict` now interprets a missing -`endian` configuration as `endian=None` (matching what `BytesCodec.to_dict` -emits), instead of falling back to the system's native byte order. diff --git a/changes/3469.bugfix.md b/changes/3469.bugfix.md deleted file mode 100644 index eb56e87476..0000000000 --- a/changes/3469.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `save_array`, `Group.__setitem__`, and `load` for 0-dimensional arrays. diff --git a/changes/3826.feature.md b/changes/3826.feature.md deleted file mode 100644 index 8909e777c5..0000000000 --- a/changes/3826.feature.md +++ /dev/null @@ -1 +0,0 @@ -Added a `subchunk_write_order` option to `ShardingCodec` to control the physical order of subchunks within a shard. Supported values are `morton`, `unordered`, `lexicographic`, and `colexicographic`. `unordered` makes no guarantee about subchunk layout. This setting affects only on-disk layout, not the data read back, and is not persisted in array metadata: it applies per codec instance and is not recovered when reopening a sharded array. diff --git a/changes/3885.bugfix.md b/changes/3885.bugfix.md deleted file mode 100644 index d4c1dd5c03..0000000000 --- a/changes/3885.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed inner-codec spec evolution for sharded arrays. The sharding codec now threads the array spec through its inner codec chain when evolving codecs, so a codec that changes the dtype upstream of `BytesCodec` no longer leaves the inner chain evolved against the wrong spec (which previously failed at decode time). This runs on the default `BatchedCodecPipeline` as well. Standard inner chains (`[BytesCodec]`, `[BytesCodec, ZstdCodec]`, transpose + bytes) are byte-identical to before. Restores the behavior of #2179. diff --git a/changes/3885.feature.1.md b/changes/3885.feature.1.md deleted file mode 100644 index 0a51ba6844..0000000000 --- a/changes/3885.feature.1.md +++ /dev/null @@ -1 +0,0 @@ -Added `SyncByteGetter` and `SyncByteSetter` runtime-checkable protocols and a `get_ranges_sync` method on the `Store` ABC. These let custom byte getters/setters opt into the synchronous codec pipeline's fast path for in-memory IO, which the sharding codec uses for its inner chunks. diff --git a/changes/3885.feature.md b/changes/3885.feature.md deleted file mode 100644 index 13010f521e..0000000000 --- a/changes/3885.feature.md +++ /dev/null @@ -1 +0,0 @@ -Added `FusedCodecPipeline`, an opt-in codec pipeline that runs codec compute synchronously and in bulk (avoiding the per-chunk async scheduling overhead of the default `BatchedCodecPipeline`), giving large speedups for sharded arrays (up to ~24x writes / ~14x reads on many-chunks-per-shard layouts, more with compression) and no regressions on compute-bound workloads. The default `BatchedCodecPipeline` is unchanged for standard configurations, so existing code keeps working unless you opt in; enable the new pipeline with `zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})`. diff --git a/changes/3899.bugfix.md b/changes/3899.bugfix.md deleted file mode 100644 index e2b2177a39..0000000000 --- a/changes/3899.bugfix.md +++ /dev/null @@ -1,7 +0,0 @@ -Make chunk normalization properly handle `-1` as a compact representation of the -length of an entire axis. Reject several previously-accepted but ill-defined -chunk specifications: `chunks=True` (previously silently produced size-1 chunks), -chunk tuples shorter than the array's number of dimensions (previously padded to -the array's shape), and `None` as a per-dimension chunk size. These all now -raise informative errors. Also fix chunk handling for 0-length array dimensions, -and add explicit rejection of 0-length chunks. diff --git a/changes/3908.misc.md b/changes/3908.misc.md deleted file mode 100644 index 66717e8444..0000000000 --- a/changes/3908.misc.md +++ /dev/null @@ -1 +0,0 @@ -Reuse a constant `ArraySpec` during indexing when possible. \ No newline at end of file diff --git a/changes/3925.feature.md b/changes/3925.feature.md deleted file mode 100644 index ed07be309c..0000000000 --- a/changes/3925.feature.md +++ /dev/null @@ -1 +0,0 @@ -Add `zarr.abc.store.Store.get_ranges` for concurrent, coalesced multi-range reads from a single key. The method is defined on the `Store` ABC with a default implementation built on `Store.get`, so every store inherits a working version; stores with native multi-range backends (e.g. `FsspecStore`) can override for efficiency. Coalescing knobs (`max_concurrency`, `max_gap_bytes`, `max_coalesced_bytes`) are passed as keyword arguments to `get_ranges`. Failures from underlying fetches surface as a `BaseExceptionGroup` (PEP 654); callers should use `except*` to filter for specific exception types such as `FileNotFoundError`. diff --git a/changes/3954.bugfix.md b/changes/3954.bugfix.md deleted file mode 100644 index 1a6c93a8ac..0000000000 --- a/changes/3954.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Handle missing consolidated metadata in leaf Group nodes. diff --git a/changes/3963.removal.md b/changes/3963.removal.md deleted file mode 100644 index a83fd53853..0000000000 --- a/changes/3963.removal.md +++ /dev/null @@ -1,6 +0,0 @@ -The ``BloscShuffle`` and ``BloscCname`` enums (``zarr.codecs.BloscShuffle``, -``zarr.codecs.BloscCname``) are now deprecated. Pass the equivalent literal -string (e.g. ``"zstd"``, ``"bitshuffle"``) when constructing a ``BloscCodec``. -The enum classes remain importable but emit ``DeprecationWarning`` on member -access, and will be removed in a future release. ``BloscCodec.cname`` and -``BloscCodec.shuffle`` are now plain strings rather than enum members. diff --git a/changes/3968.removal.md b/changes/3968.removal.md deleted file mode 100644 index 075e4a15a9..0000000000 --- a/changes/3968.removal.md +++ /dev/null @@ -1,17 +0,0 @@ -The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` -(``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the -equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / -``"end"``). The enum classes remain importable but emit ``DeprecationWarning`` -on member access, and will be removed in a future release. ``BytesCodec.endian`` -and ``ShardingCodec.index_location`` are now plain strings rather than enum -members. - -Two follow-on changes from this deprecation: - -- ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or - ``"big"``) rather than an ``Endian`` member. Subclasses overriding this - property should update their return type. -- The module-level binding ``zarr.codecs.bytes.default_system_endian`` was - removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; - external callers that imported ``default_system_endian`` should use - ``sys.byteorder`` directly. diff --git a/changes/3972.misc.md b/changes/3972.misc.md deleted file mode 100644 index 60e5f75cca..0000000000 --- a/changes/3972.misc.md +++ /dev/null @@ -1 +0,0 @@ -Run `mypy` via `uv run mypy` instead of `pre-commit`'s isolated venv. The `dev` dependency group in `pyproject.toml`, locked by `uv.lock`, is now the single source of truth for `mypy`'s dependency set, eliminating the duplicate dependency list previously maintained in `.pre-commit-config.yaml` and giving every contributor and CI an identical, reproducible type-checking environment. diff --git a/changes/3973.removal.md b/changes/3973.removal.md deleted file mode 100644 index c25ac967a2..0000000000 --- a/changes/3973.removal.md +++ /dev/null @@ -1 +0,0 @@ -Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). \ No newline at end of file diff --git a/changes/3975.misc.md b/changes/3975.misc.md deleted file mode 100644 index b90147988a..0000000000 --- a/changes/3975.misc.md +++ /dev/null @@ -1 +0,0 @@ -Store `chunks_per_shard` explicitly as a field on `_ShardIndex` instead of inferring it from `offsets_and_lengths.shape[:-1]`. The previous derivation collapsed to rank-1 for 0-D arrays, requiring a numpy-compat cast workaround that is now removed. Also removes the unused `_ShardIndex.is_dense` method, which was ported from an earlier prototype and never had any call sites or tests. diff --git a/changes/3977.bugfix.md b/changes/3977.bugfix.md deleted file mode 100644 index 6a8d9b4244..0000000000 --- a/changes/3977.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fix flaky stateful test bookkeeping when `delete_dir` matches string prefixes instead of true directory descendants. Previously a path such as `6/faNT…` could be incorrectly removed when deleting `6/f`. (See [issue #3977](https://github.com/zarr-developers/zarr-python/issues/3977).) diff --git a/changes/3979.misc.md b/changes/3979.misc.md deleted file mode 100644 index 3d49547148..0000000000 --- a/changes/3979.misc.md +++ /dev/null @@ -1 +0,0 @@ -Remove the `UnstableSpecificationWarning` emitted when serializing a `FixedLengthUTF32` data type instance, as a spec for this data type has been published to zarr-extensions. \ No newline at end of file diff --git a/changes/3984.doc.md b/changes/3984.doc.md deleted file mode 100644 index 1b92fb8e74..0000000000 --- a/changes/3984.doc.md +++ /dev/null @@ -1,4 +0,0 @@ -Clarify the difference between `zarr.load` and `zarr.open` in their docstrings. -`load` eagerly reads data into an in-memory array, while `open` returns a -lazy `Array` or `Group` backed by the store, with `See Also` cross-references -linking the two. diff --git a/changes/3987.feature.md b/changes/3987.feature.md deleted file mode 100644 index 2492b4d7dd..0000000000 --- a/changes/3987.feature.md +++ /dev/null @@ -1 +0,0 @@ -Two new fields on `ArrayConfig` control how the sharding codec coalesces partial-shard reads: `sharding_coalesce_max_gap_bytes` (default 1 MiB) and `sharding_coalesce_max_bytes` (default 16 MiB). When reading multiple chunks from the same shard, nearby byte ranges are merged into a single request to the store if separated by no more than `sharding_coalesce_max_gap_bytes` and the merged read stays within `sharding_coalesce_max_bytes`. Defaults are seeded from the matching `array.sharding_coalesce_max_gap_bytes` / `array.sharding_coalesce_max_bytes` keys in [`zarr.config`][] at array-creation time, and can be overridden per array by passing `config={...}` to [`zarr.create_array`][]. diff --git a/changes/3990.misc.md b/changes/3990.misc.md deleted file mode 100644 index ff3fcf4cf2..0000000000 --- a/changes/3990.misc.md +++ /dev/null @@ -1,5 +0,0 @@ -Widen `ChunksLike` type alias to use `Iterable` instead of `Sequence`, and also -remove `None` from the type union. This supports a broader range of types, -removing the necessity to "materialize" iterable values simply to satisfy type -annotations. It also allows use of `ChunksLike` in cases where `None` should -not be permitted. diff --git a/changes/3998.misc.md b/changes/3998.misc.md deleted file mode 100644 index bacfa93a8b..0000000000 --- a/changes/3998.misc.md +++ /dev/null @@ -1 +0,0 @@ -Centralized JSON document I/O behind free functions in `zarr.core._json` and removed the unused private `Store._get_bytes`/`_get_json` methods and their per-store overrides. diff --git a/changes/4000.misc.md b/changes/4000.misc.md deleted file mode 100644 index 17d48d3016..0000000000 --- a/changes/4000.misc.md +++ /dev/null @@ -1 +0,0 @@ -Run all doctests via pytest and fix all broken doctests. diff --git a/changes/4001.misc.md b/changes/4001.misc.md deleted file mode 100644 index adbba988d9..0000000000 --- a/changes/4001.misc.md +++ /dev/null @@ -1,6 +0,0 @@ -Restore sharding write performance for shards with many inner chunks. The -`subchunk_write_order` feature inadvertently rebuilt the per-shard chunk -coordinate grid (up to tens of thousands of coordinate tuples) on every shard -write. These coordinates are now computed once per shard shape and cached, so -repeated writes to same-shaped shards reuse them, restoring write throughput to -its previous level. diff --git a/changes/4003.bugfix.md b/changes/4003.bugfix.md deleted file mode 100644 index 36327b55df..0000000000 --- a/changes/4003.bugfix.md +++ /dev/null @@ -1,19 +0,0 @@ -`FsspecStore.from_url()` and `from_mapper()` now close the async filesystem -they create when `store.close()` is called. Previously the underlying aiohttp -`ClientSession` was left open until garbage collection, producing -`"Unclosed client session"` `ResourceWarning`s from aiohttp. - -The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when -`FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` -when a sync→async conversion was performed). When `_owns_fs` is ``True``, -`store.close()` calls the new `_close_fs()` helper, which invokes -`fs.set_session()` and closes the returned client. Callers who supply their own -filesystem instance to `FsspecStore()` directly remain responsible for its -lifecycle; `_owns_fs` is ``False`` for those stores. - -**Scope note**: This fix closes the S3 client session that is active at the time -`store.close()` is called. Some S3-backed filesystem implementations (e.g. -s3fs with ``cache_regions=True``) may internally refresh and replace their -client during I/O operations, abandoning prior sessions before ``store.close()`` -is invoked. Those intermediate sessions are outside the scope of this fix and -are an issue in the upstream filesystem library. diff --git a/changes/4016.bugfix.md b/changes/4016.bugfix.md deleted file mode 100644 index 01984110f7..0000000000 --- a/changes/4016.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. diff --git a/changes/4032.bugfix.md b/changes/4032.bugfix.md deleted file mode 100644 index 4c9476f583..0000000000 --- a/changes/4032.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. diff --git a/changes/4046.misc.md b/changes/4046.misc.md deleted file mode 100644 index 96c0c7d78a..0000000000 --- a/changes/4046.misc.md +++ /dev/null @@ -1 +0,0 @@ -Replace the internal `DefaultFillValue` sentinel class with a `typing_extensions.Sentinel`, and raise the minimum `typing_extensions` version to 4.14. diff --git a/changes/4052.doc.md b/changes/4052.doc.md deleted file mode 100644 index 90b4679db5..0000000000 --- a/changes/4052.doc.md +++ /dev/null @@ -1,21 +0,0 @@ -Updated the custom dtype example in `examples/custom_dtype/custom_dtype.py` to -use only the public API, eliminating all non-public imports, illustrating what -users should do. - -To better support this, the following types and functions were made available -from public modules: - -| Type/Function | Non-public module | Public module | -| ------------------------- | ------------------------ | ------------- | -| `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | -| `JSON` | `zarr.core.common` | `zarr.types` | -| `ZarrFormat` | `zarr.core.common` | `zarr.types` | -| `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | -| `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | -| `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | -| `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | - -`DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from -`zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or -`zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining -types and functions are simply re-exported from the listed public module. diff --git a/changes/4053.doc.md b/changes/4053.doc.md deleted file mode 100644 index 3220b2b4ec..0000000000 --- a/changes/4053.doc.md +++ /dev/null @@ -1 +0,0 @@ -Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. diff --git a/changes/4054.misc.md b/changes/4054.misc.md deleted file mode 100644 index 4aa9435b84..0000000000 --- a/changes/4054.misc.md +++ /dev/null @@ -1 +0,0 @@ -Add Hypothesis property tests for block and mask indexing (`test_block_indexing`, `test_mask_indexing`), along with a `block_indices` strategy in `zarr.testing.strategies`. These extend the existing randomized indexing coverage (basic, orthogonal, and vectorized) to the block and mask selection methods. diff --git a/changes/4059.bugfix.md b/changes/4059.bugfix.md deleted file mode 100644 index 16fb582f65..0000000000 --- a/changes/4059.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference diff --git a/changes/4073.misc.md b/changes/4073.misc.md deleted file mode 100644 index bcc6b8281e..0000000000 --- a/changes/4073.misc.md +++ /dev/null @@ -1 +0,0 @@ -Extend the `test_block_indexing` Hypothesis property test to cover rectilinear chunk grids and sharded regular grids, and generalize the `block_indices` strategy in `zarr.testing.strategies` to build its array-space oracle from cumulative chunk offsets (`chunk_sizes` parameter) instead of a uniform chunk size. diff --git a/changes/4074.bugfix.md b/changes/4074.bugfix.md deleted file mode 100644 index d55a52b887..0000000000 --- a/changes/4074.bugfix.md +++ /dev/null @@ -1,7 +0,0 @@ -Fixed several storage and codec bugs: - -- Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. -- `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). -- `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). -- `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). -- `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. diff --git a/changes/4086.misc.md b/changes/4086.misc.md deleted file mode 100644 index 2564cf9ea1..0000000000 --- a/changes/4086.misc.md +++ /dev/null @@ -1 +0,0 @@ -Fix `test_multiprocessing[fork]` failing on Python 3.15, where `os.fork()` in a multi-threaded process emits a `DeprecationWarning` that the test suite promotes to an error. diff --git a/changes/4116.bugfix.md b/changes/4116.bugfix.md deleted file mode 100644 index e99eac8c95..0000000000 --- a/changes/4116.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. diff --git a/changes/4132.doc.md b/changes/4132.doc.md deleted file mode 100644 index 3325c82f43..0000000000 --- a/changes/4132.doc.md +++ /dev/null @@ -1,6 +0,0 @@ -Fixed many documentation errors found in a full review of the user guide, including -prose contradicted by rendered example output on the performance page, invisible -code blocks, an incorrect S3 example, stale "not yet implemented" claims in the -v3 migration guide, and undocumented optional dependency groups. Also improved -navigation order, cross-linking between pages, and coverage of group member -enumeration, bulk attribute updates, and the `use_consolidated` keyword. diff --git a/changes/4133.doc.md b/changes/4133.doc.md deleted file mode 100644 index 9690318b46..0000000000 --- a/changes/4133.doc.md +++ /dev/null @@ -1 +0,0 @@ -Fixed the documented default of ``max_age_seconds`` in the ``CacheStore`` docstring: the default is ``"infinity"`` (no expiration), not ``None``, which is rejected. Also noted that ``cache_store`` must support deletes. diff --git a/changes/4138.misc.md b/changes/4138.misc.md deleted file mode 100644 index 2cf0365bb9..0000000000 --- a/changes/4138.misc.md +++ /dev/null @@ -1 +0,0 @@ -Pass a list rather than a generator to `pytest.mark.parametrize` in `tests/test_docs.py`, so the test suite collects cleanly under pytest 9.1 (which deprecates non-Collection iterables and, under `filterwarnings = error`, turns that into a collection error). diff --git a/changes/4141.bugfix.md b/changes/4141.bugfix.md deleted file mode 100644 index 6a132da3f5..0000000000 --- a/changes/4141.bugfix.md +++ /dev/null @@ -1 +0,0 @@ -Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. diff --git a/docs/release-notes.md b/docs/release-notes.md index 3ba870ac92..e85137f2b6 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -2,6 +2,162 @@ +## 3.3.0 (2026-07-15) + +### Features + +- Optimizes reading multiple chunks from a shard. Serial calls to `Store.get()` + in the sharding codec have been replaced with a single call to + `Store.get_ranges()`, which coalesces nearby byte ranges and fetches them + concurrently. ([#3004](https://github.com/zarr-developers/zarr-python/issues/3004)) +- Added a `subchunk_write_order` option to `ShardingCodec` to control the physical order of subchunks within a shard. Supported values are `morton`, `unordered`, `lexicographic`, and `colexicographic`. `unordered` makes no guarantee about subchunk layout. This setting affects only on-disk layout, not the data read back, and is not persisted in array metadata: it applies per codec instance and is not recovered when reopening a sharded array. ([#3826](https://github.com/zarr-developers/zarr-python/issues/3826)) +- Added `SyncByteGetter` and `SyncByteSetter` runtime-checkable protocols and a `get_ranges_sync` method on the `Store` ABC. These let custom byte getters/setters opt into the synchronous codec pipeline's fast path for in-memory IO, which the sharding codec uses for its inner chunks. ([#3885](https://github.com/zarr-developers/zarr-python/issues/3885)) +- Added `FusedCodecPipeline`, an opt-in codec pipeline that runs codec compute synchronously and in bulk (avoiding the per-chunk async scheduling overhead of the default `BatchedCodecPipeline`), giving large speedups for sharded arrays (up to ~24x writes / ~14x reads on many-chunks-per-shard layouts, more with compression) and no regressions on compute-bound workloads. The default `BatchedCodecPipeline` is unchanged for standard configurations, so existing code keeps working unless you opt in; enable the new pipeline with `zarr.config.set({"codec_pipeline.path": "zarr.core.codec_pipeline.FusedCodecPipeline"})`. ([#3885](https://github.com/zarr-developers/zarr-python/issues/3885)) +- Add `zarr.abc.store.Store.get_ranges` for concurrent, coalesced multi-range reads from a single key. The method is defined on the `Store` ABC with a default implementation built on `Store.get`, so every store inherits a working version; stores with native multi-range backends (e.g. `FsspecStore`) can override for efficiency. Coalescing knobs (`max_concurrency`, `max_gap_bytes`, `max_coalesced_bytes`) are passed as keyword arguments to `get_ranges`. Failures from underlying fetches surface as a `BaseExceptionGroup` (PEP 654); callers should use `except*` to filter for specific exception types such as `FileNotFoundError`. ([#3925](https://github.com/zarr-developers/zarr-python/issues/3925)) +- Two new fields on `ArrayConfig` control how the sharding codec coalesces partial-shard reads: `sharding_coalesce_max_gap_bytes` (default 1 MiB) and `sharding_coalesce_max_bytes` (default 16 MiB). When reading multiple chunks from the same shard, nearby byte ranges are merged into a single request to the store if separated by no more than `sharding_coalesce_max_gap_bytes` and the merged read stays within `sharding_coalesce_max_bytes`. Defaults are seeded from the matching `array.sharding_coalesce_max_gap_bytes` / `array.sharding_coalesce_max_bytes` keys in [`zarr.config`][] at array-creation time, and can be overridden per array by passing `config={...}` to [`zarr.create_array`][]. ([#3987](https://github.com/zarr-developers/zarr-python/issues/3987)) + +### Bugfixes + +- Stop emitting an `UnstableSpecificationWarning` when serializing the `struct` data type to Zarr V3 metadata. The `struct` data type now has a stable Zarr V3 specification. The legacy `structured` alias and the unspecified `null_terminated_bytes`, `raw_bytes`, and `variable_length_bytes` data types continue to warn. ([#202](https://github.com/zarr-developers/zarr-python/issues/202)) +- Fix equality comparison of `ArrayV2Metadata` and `ArrayV3Metadata` objects with a + `NaN` fill value. Such objects are now compared by their JSON-serialized form, so two + otherwise-identical metadata objects with a `NaN` (or infinite) fill value compare equal. ([#2929](https://github.com/zarr-developers/zarr-python/issues/2929)) +- Fixed `BytesCodec.from_dict` so that `BytesCodec` instances roundtrip to / from + their dict representation. `BytesCodec.from_dict` now interprets a missing + `endian` configuration as `endian=None` (matching what `BytesCodec.to_dict` + emits), instead of falling back to the system's native byte order. ([#3417](https://github.com/zarr-developers/zarr-python/issues/3417)) +- Fixed `save_array`, `Group.__setitem__`, and `load` for 0-dimensional arrays. ([#3469](https://github.com/zarr-developers/zarr-python/issues/3469)) +- Fixed inner-codec spec evolution for sharded arrays. The sharding codec now threads the array spec through its inner codec chain when evolving codecs, so a codec that changes the dtype upstream of `BytesCodec` no longer leaves the inner chain evolved against the wrong spec (which previously failed at decode time). This runs on the default `BatchedCodecPipeline` as well. Standard inner chains (`[BytesCodec]`, `[BytesCodec, ZstdCodec]`, transpose + bytes) are byte-identical to before. Restores the behavior of #2179. ([#3885](https://github.com/zarr-developers/zarr-python/issues/3885)) +- Make chunk normalization properly handle `-1` as a compact representation of the + length of an entire axis. Reject several previously-accepted but ill-defined + chunk specifications: `chunks=True` (previously silently produced size-1 chunks), + chunk tuples shorter than the array's number of dimensions (previously padded to + the array's shape), and `None` as a per-dimension chunk size. These all now + raise informative errors. Also fix chunk handling for 0-length array dimensions, + and add explicit rejection of 0-length chunks. ([#3899](https://github.com/zarr-developers/zarr-python/issues/3899)) +- Handle missing consolidated metadata in leaf Group nodes. ([#3954](https://github.com/zarr-developers/zarr-python/issues/3954)) +- Corrected the JSON type definitions for the `numpy.datetime64` and + `numpy.timedelta64` data types in Zarr V3 metadata: the `configuration` object + (holding `unit` and `scale_factor`) is now required, matching the published + specifications for these data types. Also updated the specification links in + the docstrings to point to the zarr-extensions repository. ([#3955](https://github.com/zarr-developers/zarr-python/issues/3955)) +- Fixed writing to 0-dimensional arrays that use the sharding codec. Previously + assigning to a 0-dimensional sharded array raised an error. ([#3966](https://github.com/zarr-developers/zarr-python/issues/3966)) +- Fix flaky stateful test bookkeeping when `delete_dir` matches string prefixes instead of true directory descendants. Previously a path such as `6/faNT…` could be incorrectly removed when deleting `6/f`. (See [issue #3977](https://github.com/zarr-developers/zarr-python/issues/3977).) ([#3977](https://github.com/zarr-developers/zarr-python/issues/3977)) +- `FsspecStore.from_url()` and `from_mapper()` now close the async filesystem + they create when `store.close()` is called. Previously the underlying aiohttp + `ClientSession` was left open until garbage collection, producing + `"Unclosed client session"` `ResourceWarning`s from aiohttp. + + The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when + `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` + when a sync→async conversion was performed). When `_owns_fs` is ``True``, + `store.close()` calls the new `_close_fs()` helper, which invokes + `fs.set_session()` and closes the returned client. Callers who supply their own + filesystem instance to `FsspecStore()` directly remain responsible for its + lifecycle; `_owns_fs` is ``False`` for those stores. + + **Scope note**: This fix closes the S3 client session that is active at the time + `store.close()` is called. Some S3-backed filesystem implementations (e.g. + s3fs with ``cache_regions=True``) may internally refresh and replace their + client during I/O operations, abandoning prior sessions before ``store.close()`` + is invoked. Those intermediate sessions are outside the scope of this fix and + are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) +- Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. ([#4016](https://github.com/zarr-developers/zarr-python/issues/4016)) +- Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. ([#4032](https://github.com/zarr-developers/zarr-python/issues/4032)) +- Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference ([#4059](https://github.com/zarr-developers/zarr-python/issues/4059)) +- Fixed several storage and codec bugs: + + - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. + - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). + - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). + - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). + - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + + ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) +- Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. ([#4116](https://github.com/zarr-developers/zarr-python/issues/4116)) +- Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. ([#4141](https://github.com/zarr-developers/zarr-python/issues/4141)) + +### Improved Documentation + +- Document the changes to `zarr.errors` in the 3.0 migration guide, including the removal of v2 exception classes and the introduction of `NodeNotFoundError`. ([#3009](https://github.com/zarr-developers/zarr-python/issues/3009)) +- Clarify the difference between `zarr.load` and `zarr.open` in their docstrings. + `load` eagerly reads data into an in-memory array, while `open` returns a + lazy `Array` or `Group` backed by the store, with `See Also` cross-references + linking the two. ([#3984](https://github.com/zarr-developers/zarr-python/issues/3984)) +- Updated the custom dtype example in `examples/custom_dtype/custom_dtype.py` to + use only the public API, eliminating all non-public imports, illustrating what + users should do. + + To better support this, the following types and functions were made available + from public modules: + + | Type/Function | Non-public module | Public module | + | ------------------------- | ------------------------ | ------------- | + | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | + | `JSON` | `zarr.core.common` | `zarr.types` | + | `ZarrFormat` | `zarr.core.common` | `zarr.types` | + | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | + | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | + + `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from + `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or + `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining + types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) +- Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. ([#4053](https://github.com/zarr-developers/zarr-python/issues/4053)) +- Fixed many documentation errors found in a full review of the user guide, including + prose contradicted by rendered example output on the performance page, invisible + code blocks, an incorrect S3 example, stale "not yet implemented" claims in the + v3 migration guide, and undocumented optional dependency groups. Also improved + navigation order, cross-linking between pages, and coverage of group member + enumeration, bulk attribute updates, and the `use_consolidated` keyword. ([#4132](https://github.com/zarr-developers/zarr-python/issues/4132)) +- Fixed the documented default of ``max_age_seconds`` in the ``CacheStore`` docstring: the default is ``"infinity"`` (no expiration), not ``None``, which is rejected. Also noted that ``cache_store`` must support deletes. ([#4133](https://github.com/zarr-developers/zarr-python/issues/4133)) + +### Deprecations and Removals + +- The ``BloscShuffle`` and ``BloscCname`` enums (``zarr.codecs.BloscShuffle``, + ``zarr.codecs.BloscCname``) are now deprecated. Pass the equivalent literal + string (e.g. ``"zstd"``, ``"bitshuffle"``) when constructing a ``BloscCodec``. + The enum classes remain importable but emit ``DeprecationWarning`` on member + access, and will be removed in a future release. ``BloscCodec.cname`` and + ``BloscCodec.shuffle`` are now plain strings rather than enum members. + + Additional renames in ``zarr.codecs.blosc`` from the same change: the type + aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and + ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` + (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` + now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) +- The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` + (``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the + equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / + ``"end"``). The enum classes remain importable but emit ``DeprecationWarning`` + on member access, and will be removed in a future release. ``BytesCodec.endian`` + and ``ShardingCodec.index_location`` are now plain strings rather than enum + members. + + Two follow-on changes from this deprecation: + + - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or + ``"big"``) rather than an ``Endian`` member. Subclasses overriding this + property should update their return type. + - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was + removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; + external callers that imported ``default_system_endian`` should use + ``sys.byteorder`` directly. + + Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` + was made private as part of this change. + + ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) +- Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). ([#3973](https://github.com/zarr-developers/zarr-python/issues/3973)) + +### Misc + +- [#214](https://github.com/zarr-developers/zarr-python/issues/214), [#215](https://github.com/zarr-developers/zarr-python/issues/215), [#3908](https://github.com/zarr-developers/zarr-python/issues/3908), [#3972](https://github.com/zarr-developers/zarr-python/issues/3972), [#3975](https://github.com/zarr-developers/zarr-python/issues/3975), [#3979](https://github.com/zarr-developers/zarr-python/issues/3979), [#3990](https://github.com/zarr-developers/zarr-python/issues/3990), [#3998](https://github.com/zarr-developers/zarr-python/issues/3998), [#4000](https://github.com/zarr-developers/zarr-python/issues/4000), [#4001](https://github.com/zarr-developers/zarr-python/issues/4001), [#4046](https://github.com/zarr-developers/zarr-python/issues/4046), [#4054](https://github.com/zarr-developers/zarr-python/issues/4054), [#4073](https://github.com/zarr-developers/zarr-python/issues/4073), [#4086](https://github.com/zarr-developers/zarr-python/issues/4086), [#4138](https://github.com/zarr-developers/zarr-python/issues/4138) + + ## 3.2.1 (2026-05-05) ### Bugfixes From 23d96d38ae1cb1b722681f16eec7af171890b367 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Thu, 16 Jul 2026 11:30:12 -0400 Subject: [PATCH 05/16] chore: CI worfklows for linting docs (#4076) * chore: CI worfklows for missing docs and RST docstrings * Add linter for docs * Add link checking workflow * fix links * fix: apply suggestions from code review * refactor: readability improvements * chore: switch lychee action to opening issue on failure --------- Co-authored-by: Davis Bennett --- .github/workflows/docs.yml | 5 + .github/workflows/links.yml | 32 +++ .markdownlint-cli2.jsonc | 54 +++++ .pre-commit-config.yaml | 10 + ci/check_documented_exports.py | 161 +++++++++++++ ci/lint_docs.py | 284 +++++++++++++++++++++++ docs/contributing.md | 23 +- docs/index.md | 14 +- docs/quick-start.md | 2 - docs/release-notes.md | 2 + docs/user-guide/attributes.md | 1 + docs/user-guide/consolidated_metadata.md | 1 - docs/user-guide/data_types.md | 7 + docs/user-guide/experimental.md | 1 - docs/user-guide/extending.md | 1 + docs/user-guide/groups.md | 1 - docs/user-guide/performance.md | 6 +- docs/user-guide/storage.md | 8 +- lychee.toml | 20 ++ src/zarr/core/metadata/v2.py | 2 +- src/zarr/testing/store.py | 6 +- 21 files changed, 608 insertions(+), 33 deletions(-) create mode 100644 .github/workflows/links.yml create mode 100644 .markdownlint-cli2.jsonc create mode 100644 ci/check_documented_exports.py create mode 100644 ci/lint_docs.py create mode 100644 lychee.toml diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index 59eca8d9b9..c72b493b12 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -24,6 +24,11 @@ jobs: persist-credentials: false - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - run: uv sync --group docs + # Fast source-level guards that need no built site, so they run before the (slower) + # build for a quick failure: every public export is in the API reference, and no + # docstring/Markdown carries reStructuredText markup that MkDocs won't render. + - run: uv run python ci/check_documented_exports.py docs/api + - run: uv run python ci/lint_docs.py # --strict turns warnings into errors, so a docs code block that fails to execute # at build time (e.g. a non-exec python fence disrupting a later exec="true" block) # fails CI instead of merging as a silent warning. diff --git a/.github/workflows/links.yml b/.github/workflows/links.yml new file mode 100644 index 0000000000..a53de8da83 --- /dev/null +++ b/.github/workflows/links.yml @@ -0,0 +1,32 @@ +name: Check links + +on: + repository_dispatch: + workflow_dispatch: + # pull_request: + schedule: + - cron: "00 18 * * *" + +jobs: + linkChecker: + runs-on: ubuntu-latest + permissions: + issues: write # required for peter-evans/create-issue-from-file + steps: + - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + with: + persist-credentials: false + + - name: Link Checker + id: lychee + uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2.8.0 + with: + fail: false + + - name: Create Issue From File + if: steps.lychee.outputs.exit_code != 0 + uses: peter-evans/create-issue-from-file@fca9117c27cdc29c6c4db3b86c48e4115a786710 # v6.0.0 + with: + title: Link Checker Report + content-filepath: ./lychee/out.md + labels: report, automated issue diff --git a/.markdownlint-cli2.jsonc b/.markdownlint-cli2.jsonc new file mode 100644 index 0000000000..3dfdf96856 --- /dev/null +++ b/.markdownlint-cli2.jsonc @@ -0,0 +1,54 @@ +// markdownlint-cli2 configuration for zarr-python docs. +// +// We keep the rules that catch real rendering/structure problems and disable those that +// are pure style, conflict with house conventions, or fire false positives against our +// MkDocs/mkdocstrings + pymdownx toolchain. Complementary, not overlapping, with +// ci/lint_docs.py (RST residue + list-breaking fences) and `mkdocs build --strict`. +{ + "config": { + "default": true, + + // House style: Markdown paragraphs are single unwrapped lines, so line length is not + // a meaningful constraint. + "MD013": false, + + // Purely stylistic marker/emphasis choices -- not worth the churn across existing docs. + "MD004": false, // ul bullet style (-, *, +) + "MD007": false, // ul indentation width + "MD050": false, // strong (bold) style + "MD035": false, // hr style + + // False positives from our toolchain: + // mkdocstrings cross-refs `[`X`][zarr.X]` read as undefined reference links (MD052); + // pymdownx.magiclink auto-links bare URLs (MD034); + // md_in_html lets us embed intentional raw HTML (MD033); + // generated/included files (api stubs, snippets) need not open with an H1 (MD041). + "MD052": false, + "MD034": false, + "MD033": false, + "MD041": false, + + // Duplicate headings are legitimate under different sections (e.g. repeated + // "Documentation"); only flag true sibling duplicates. + "MD024": { "siblings_only": true }, + + // Opinionated table/link/command rules with low value for these docs. + "MD055": false, // table pipe style + "MD060": false, // table column style + "MD059": false, // "descriptive" link text (no "click here") + "MD014": false, // $ before commands without shown output + + // markdownlint does not understand MkDocs `!!!` admonitions, so it reads their + // 4-space-indented bodies as indented code blocks and flags them (and, via inferred + // file style, flags real fenced blocks too). Cannot coexist with our admonitions. + "MD046": false // code block style (fenced vs indented) + // Kept on (structural / real rendering bugs): MD012 (multiple blanks), MD022/MD031/MD032 + // (blanks around headings/fences/lists), MD025 (single H1), MD029 (ordered-list prefix), + // MD040 (fenced code language), MD042 (empty links), + // MD047 (trailing newline), MD056 (table column count), among others. + }, + "globs": ["docs/**/*.md"], + "ignores": [ + "docs/api/**" // mkdocstrings stubs (`::: zarr.X`) + ] +} diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index e08c98abca..57a1d0d4f7 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -31,6 +31,16 @@ repos: - id: check-yaml exclude: mkdocs.yml - id: trailing-whitespace + - repo: https://github.com/DavidAnson/markdownlint-cli2 + rev: v0.22.1 + hooks: + # Markdown structure/hygiene. Rule selection and ignores are in + # .markdownlint-cli2.jsonc; complements ci/lint_docs.py (RST residue, + # list-breaking fences) and `mkdocs build --strict`. Scoped to docs/ to + # match the config's globs (pre-commit passes filenames, which would + # otherwise override that scoping and lint all repo Markdown). + - id: markdownlint-cli2 + files: ^docs/ - repo: local hooks: - id: mypy diff --git a/ci/check_documented_exports.py b/ci/check_documented_exports.py new file mode 100644 index 0000000000..0772954399 --- /dev/null +++ b/ci/check_documented_exports.py @@ -0,0 +1,161 @@ +"""Check that every public top-level export is in the API reference. + +The API reference is authored as explicit mkdocstrings directives (``::: target``) +under ``docs/api/`` -- one per documented symbol -- rather than autodoc, so a newly +added ``zarr.__all__`` entry will not appear in the docs until someone writes a page +for it (or it becomes a rendered member of an already-documented module). This script +catches that gap: it resolves every ``:::`` target, expands module directives into the +members they render (honoring ``members: false``), and asserts each name in +``zarr.__all__`` resolves to a documented object. + +Usage: + python ci/check_documented_exports.py [API_DOCS_DIR] + +API_DOCS_DIR defaults to the repo-root ``docs/api``. Exits non-zero (and prints the +undocumented exports to stderr) if any public export is missing from the reference. +""" + +from __future__ import annotations + +import importlib +import re +import sys +from pathlib import Path +from types import ModuleType +from typing import TYPE_CHECKING, Any + +import zarr + +if TYPE_CHECKING: + from collections.abc import Iterator + +REPO_ROOT = Path(__file__).parent.parent.resolve() +DEFAULT_API_DOCS_ROOT = REPO_ROOT / "docs" / "api" + +# Names in zarr.__all__ that are intentionally absent from the API reference. +# Keep this list short and justified -- it is the only escape hatch from the guard. +EXEMPT_EXPORTS = { + "__version__", # version string, not an API symbol + "print_debug_info", # debugging helper, deliberately not in the reference +} + +# A mkdocstrings autodoc directive: `::: some.dotted.target` at the start of a line. +DIRECTIVE_RE = re.compile(r"^:::[ \t]+(?P\S+)") +# `members: false` (or `members: []`) within a directive's option block disables +# rendering of a module's members. +MEMBERS_DISABLED_RE = re.compile(r"^\s+members:\s*(false|\[\s*\])\s*$") + + +def resolve(target: str) -> Any: + """Resolve a `:::` target (a dotted path) to the Python object it documents.""" + try: + return importlib.import_module(target) + except ImportError: + pass + module_path, _, attr = target.rpartition(".") + try: + return getattr(importlib.import_module(module_path), attr) + except (ImportError, AttributeError): + return None + + +def iter_directives(text: str) -> Iterator[tuple[str, bool]]: + """Yield ``(target, members_enabled)`` for each ``:::`` directive in ``text``. + + The file is split into lines once; for each directive we scan its indented option + block -- stopping at the first non-indented line, which ends the block -- so options + belonging to a later directive are never consulted. ``members_enabled`` is False when + that block sets ``members: false`` (or ``members: []``).""" + lines = text.splitlines() + i = 0 + while i < len(lines): + match = DIRECTIVE_RE.match(lines[i]) + if match is None: + i += 1 + continue + members_enabled = True + i += 1 + while i < len(lines): + line = lines[i] + if line.strip() == "": + i += 1 + continue + if not line.startswith((" ", "\t")): + break # non-indented line: end of this directive's option block + if MEMBERS_DISABLED_RE.match(line): + members_enabled = False + i += 1 + yield match.group("target"), members_enabled + + +def module_member_ids(module: ModuleType) -> Iterator[int]: + """Yield the id() of each public member a module directive renders. + + The rendered members are the module's ``__all__`` if defined, else its public + (non-underscore) attributes.""" + member_names = getattr(module, "__all__", None) or [ + name for name in dir(module) if not name.startswith("_") + ] + for name in member_names: + member = getattr(module, name, None) + if member is not None: + yield id(member) + + +def documented_object_ids(api_docs_root: Path) -> set[int]: + """Collect the id()s of every object rendered by a `:::` directive under api_docs_root. + + A directive pointing at an object documents that object. A directive pointing at a + module documents the module's public members unless the directive sets + ``members: false``.""" + documented: set[int] = set() + for md_file in sorted(api_docs_root.rglob("*.md")): + for target, members_enabled in iter_directives(md_file.read_text(encoding="utf-8")): + obj = resolve(target) + if obj is None: + continue + documented.add(id(obj)) + if isinstance(obj, ModuleType) and members_enabled: + documented.update(module_member_ids(obj)) + return documented + + +def find_undocumented_exports(api_docs_root: Path) -> list[str]: + documented = documented_object_ids(api_docs_root) + return sorted( + name + for name in zarr.__all__ + if name not in EXEMPT_EXPORTS and id(getattr(zarr, name)) not in documented + ) + + +def main() -> int: + args = sys.argv[1:] + api_docs_root = Path(args[0]).resolve() if args else DEFAULT_API_DOCS_ROOT + if not api_docs_root.exists(): + print(f"{api_docs_root} does not exist.", file=sys.stderr) + return 1 + + missing = find_undocumented_exports(api_docs_root) + if not missing: + print(f"All {len(zarr.__all__)} public exports are documented.") + return 0 + + print( + f"Found {len(missing)} public export(s) in zarr.__all__ missing from the API " + "reference (docs/api/):\n", + file=sys.stderr, + ) + for name in missing: + print(f" - zarr.{name}", file=sys.stderr) + print( + "\nAdd a `::: zarr.` page under docs/api/zarr/ (and register it in " + "mkdocs.yml and docs/api/zarr/index.md), or -- if the export is intentionally " + "undocumented -- add it to EXEMPT_EXPORTS in this script with a reason.", + file=sys.stderr, + ) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/ci/lint_docs.py b/ci/lint_docs.py new file mode 100644 index 0000000000..ff7b56649b --- /dev/null +++ b/ci/lint_docs.py @@ -0,0 +1,284 @@ +"""Lint docstrings and Markdown for reStructuredText markup that won't render. + +This project renders API docs with mkdocstrings (``docstring_style: numpy``) and prose +with MkDocs + Markdown -- not Sphinx/reStructuredText. RST constructs that survive from +older docstrings (or muscle memory) are not interpreted: a Sphinx role passes through as +literal text instead of becoming a link, an ``.. note::`` directive renders as a stray +line, and a ``:param:`` field list never becomes a documented parameter. + +Crucially, none of this is caught by the rest of the docs CI. ``mkdocs build --strict`` +sees the residue as ordinary prose (no warning), and ``ci/check_unlinked_types.py`` only +finds cross-references mkdocstrings *attempted* to resolve -- a raw ``:class:`` role is +never attempted, so it leaves no unlinked-type span. This linter fills that gap with a +fast, source-level check that needs no docs build. + +Checks fall into two groups -- RST markup that silently fails under MkDocs/mkdocstrings, +and a Markdown structural problem that renders as valid-but-wrong HTML (so `mkdocs build` +emits no warning): + + sphinx-role :class:`X`, :func:`X`, :py:meth:`X` -> [`X`][zarr.X] + rst-directive .. note:: / .. code-block:: python -> MkDocs admonition / fenced code + rst-field :param x:, :returns:, :rtype: -> numpydoc Parameters/Returns/Raises + rst-link `text `_ -> [text](https://example) + list-break unindented code fence between list items -> indent the fence under its item + +The ``list-break`` check catches a fenced code block at column 0 placed *between* two list +items: because the fence is not indented into the preceding item, Markdown ends the list at +the fence and the following item starts a fresh list -- renumbering an ordered list (1, 1, 2 +instead of 1, 2, 3) or breaking the grouping/spacing of any list. markdownlint's MD029 only +notices this for sequentially-numbered ordered lists; lazily-numbered (1., 1.) and unordered +lists slip past it, so this structural check covers the gap. + +Usage: + python ci/lint_docs.py [PATH ...] + +PATH defaults to the repo-root ``src/zarr`` and ``docs``. Each PATH may be a file or a +directory (directories are searched for ``*.py`` and ``*.md``). Exits non-zero if any +issues are found. +""" + +from __future__ import annotations + +import ast +import re +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import NamedTuple + +REPO_ROOT = Path(__file__).parent.parent.resolve() +DEFAULT_PATHS = (REPO_ROOT / "src" / "zarr", REPO_ROOT / "docs") + +# A Sphinx interpreted-text role: an optional domain, a role name, then a backtick +# target -- e.g. :class:`Foo` or :py:meth:`Foo.bar`. Requires the trailing backtick so +# plain "::" (RST literal markers, time strings, mkdocs-material :icon: shortcodes) and +# URLs ("https://") never match. +SPHINX_ROLE = re.compile(r":[a-zA-Z_]\w*(?::[a-zA-Z_]\w*)?:`[^`\n]+`") + +# An RST directive line: ".. name::" (with or without an argument after it). RST hyperlink +# targets (".. _label:") and comments (".. text") lack the "::" and are not flagged. +RST_DIRECTIVE = re.compile(r"^\s*\.\.[ \t]+[\w-]+::") + +# An RST field-list entry used for docstring fields. The role names above (class, func, +# ...) are deliberately excluded so a role is reported as a role, not a field. +RST_FIELD = re.compile( + r"^\s*:(param|parameter|arg|argument|key|keyword|kwarg|type|returns?|rtype" + r"|raises?|except|exception|yields?|ytype|var|cvar|ivar)\b[^:]*:" +) + +# An RST external hyperlink: `text `_ +RST_LINK = re.compile(r"`[^`\n]+\n]+>`_") + +# A list item at column 0: an ordered marker (1. / 1)) or a bullet (-, *, +) followed by +# whitespace and content. Leading-whitespace (nested/continuation) lines are intentionally +# not matched -- the list-break check only fires on top-level items. +LIST_ITEM = re.compile(r"^(?:\d+[.)]|[-*+])\s+\S") + + +class Check(NamedTuple): + """One docs-residue check: its category, the line pattern that flags it (None for a + structural check matched outside ``_scan_line``), and the user-facing remediation + shown by ``main()``. Keeping ``example``/``fix`` here makes this the single source for + the help text, so adding a check can't leave the help out of date.""" + + category: str + pattern: re.Pattern[str] | None + example: str + fix: str + + +# ``list-break`` carries no pattern -- it is detected structurally in find_list_breaking_fences, +# not by scanning a single line -- but it appears here so it shares the remediation help. +CHECKS = ( + Check("sphinx-role", SPHINX_ROLE, ":class:`X`", "[`X`][zarr.X]"), + Check("rst-directive", RST_DIRECTIVE, ".. note::", "MkDocs admonition (!!! note)"), + Check("rst-field", RST_FIELD, ":param x:", "numpydoc Parameters/Returns/Raises section"), + Check("rst-link", RST_LINK, "`text `_", "[text](url)"), + Check("list-break", None, "fence between items", "indent the fence 4 spaces to nest it"), +) + + +@dataclass(frozen=True) +class Finding: + path: Path + line: int + category: str + snippet: str + + def format(self) -> str: + try: + location: Path | str = self.path.relative_to(REPO_ROOT) + except ValueError: + location = self.path + return f" {location}:{self.line}: [{self.category}] {self.snippet.strip()}" + + +def _scan_line(text: str) -> list[str]: + """Return every RST-residue category found in a single line (a line can carry more + than one, e.g. a role and an external link).""" + return [c.category for c in CHECKS if c.pattern is not None and c.pattern.search(text)] + + +def lint_python(path: Path) -> list[Finding]: + """Scan the docstrings (module, classes, functions) of a Python file. + + Only docstrings are checked -- they are what mkdocstrings renders -- so RST-looking + text inside ordinary code or string literals is never misreported.""" + source = path.read_text(encoding="utf-8") + try: + tree = ast.parse(source) + except SyntaxError as exc: # pragma: no cover - surfaced, not silently skipped + return [Finding(path, exc.lineno or 0, "syntax-error", str(exc.msg))] + + doc_nodes = (ast.Module, ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef) + # node.body[0].value is the docstring literal; its lineno is the line the string opens + # on, so content line i maps to source line (start + i). + docstrings = [ + (docstring, node.body[0].value.lineno) # type: ignore[attr-defined] + for node in ast.walk(tree) + if isinstance(node, doc_nodes) + if (docstring := ast.get_docstring(node, clean=False)) + ] + return [ + Finding(path, start + offset, category, line) + for docstring, start in docstrings + for offset, line in enumerate(docstring.splitlines()) + for category in _scan_line(line) + ] + + +class Fence(NamedTuple): + """A fenced code block, by 0-based line index. ``terminated`` is False when the fence + has no closing delimiter before EOF, in which case ``close`` is the last line.""" + + open: int + close: int + terminated: bool + + +def fenced_blocks(lines: list[str]) -> list[Fence]: + """Index every fenced code block in ``lines``. + + An unterminated fence is malformed Markdown that `mkdocs build` surfaces anyway; it is + still returned (with ``terminated=False``, ``close`` at the last line) so callers that + skip code can skip to EOF.""" + blocks: list[Fence] = [] + fence: str | None = None + open_idx = -1 + for i, line in enumerate(lines): + stripped = line.lstrip() + if fence is None: + if stripped.startswith(("```", "~~~")): + fence, open_idx = stripped[:3], i + elif stripped.startswith(fence): + blocks.append(Fence(open_idx, i, terminated=True)) + fence = None + if fence is not None: + blocks.append(Fence(open_idx, len(lines) - 1, terminated=False)) + return blocks + + +def find_list_breaking_fences(lines: list[str], blocks: list[Fence]) -> list[tuple[int, str]]: + """Return ``(lineno, snippet)`` for each fenced code block at column 0 that splits a + list -- i.e. one whose nearest non-blank neighbours on both sides are top-level list + items. Such a fence is not indented into the preceding item, so Markdown closes the + list at the fence and the following item starts a new one. The fix is to indent the + fence (4 spaces) so it nests inside its list item. See the module docstring. + + Conservative on purpose: it requires a list item *directly* before and after (a + continuation line or paragraph in between is not matched), keeping false positives low + for a check that fails CI. Unterminated fences are ignored.""" + + def neighbour(start: int, step: int) -> str | None: + j = start + step + while 0 <= j < len(lines): + if lines[j].strip(): + return lines[j] + j += step + return None + + def splits_a_list(open_i: int, close_i: int) -> bool: + if lines[open_i][:1].isspace(): + return False # indented fence: already nested in the list item, not a break + before = neighbour(open_i, -1) + after = neighbour(close_i, +1) + return bool(before and after and LIST_ITEM.match(before) and LIST_ITEM.match(after)) + + return [ + (fence.open + 1, lines[fence.open]) + for fence in blocks + if fence.terminated and splits_a_list(fence.open, fence.close) + ] + + +def lint_markdown(path: Path) -> list[Finding]: + """Scan a Markdown file: RST residue in prose (skipping fenced code blocks), plus + fenced code blocks that break a list (see find_list_breaking_fences).""" + lines = path.read_text(encoding="utf-8").splitlines() + blocks = fenced_blocks(lines) + in_code = {i for fence in blocks for i in range(fence.open, fence.close + 1)} + + prose = [ + Finding(path, lineno, category, line) + for lineno, line in enumerate(lines, start=1) + if lineno - 1 not in in_code + for category in _scan_line(line) + ] + breaks = [ + Finding(path, lineno, "list-break", snippet) + for lineno, snippet in find_list_breaking_fences(lines, blocks) + ] + return prose + breaks + + +def iter_files(paths: tuple[Path, ...]) -> list[Path]: + files: list[Path] = [] + for path in paths: + if path.is_file(): + files.append(path) + elif path.is_dir(): + files.extend(sorted(path.rglob("*.py"))) + files.extend(sorted(path.rglob("*.md"))) + else: + raise FileNotFoundError(f"{path} does not exist") + return files + + +LINTERS = {".py": lint_python, ".md": lint_markdown} + + +def lint(paths: tuple[Path, ...]) -> list[Finding]: + return [ + finding + for file in iter_files(paths) + if file.suffix in LINTERS + for finding in LINTERS[file.suffix](file) + ] + + +def main() -> int: + args = sys.argv[1:] + paths = tuple(Path(a).resolve() for a in args) if args else DEFAULT_PATHS + findings = lint(paths) + + if not findings: + print("No reStructuredText residue or list-breaking fences found in docs.") + return 0 + + print( + f"Found {len(findings)} docs issue(s) -- RST markup that will not render under " + "MkDocs/mkdocstrings, or Markdown that renders as valid-but-wrong HTML:\n", + file=sys.stderr, + ) + for finding in findings: + print(finding.format(), file=sys.stderr) + remediation = "\n".join(f" {c.category:<13} {c.example:<19} -> {c.fix}" for c in CHECKS) + print( + f"\nFix each issue (see ci/lint_docs.py header):\n{remediation}", + file=sys.stderr, + ) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/docs/contributing.md b/docs/contributing.md index eaab26fbc7..df46f381de 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -12,23 +12,23 @@ If you find a bug, please raise a [GitHub issue](https://github.com/zarr-develop 1. A minimal, self-contained snippet of Python code reproducing the problem. You can format the code nicely using markdown, e.g.: -```python exec="false" reason="illustrative pseudocode with a '# etc.' placeholder, not runnable" -import zarr -g = zarr.group() -# etc. -``` + ```python exec="false" reason="illustrative pseudocode with a '# etc.' placeholder, not runnable" + import zarr + g = zarr.group() + # etc. + ``` 2. An explanation of why the current behavior is wrong/not desired, and what you expect instead. 3. Information about the version of Zarr, along with versions of dependencies and the Python interpreter, and installation information. The version of Zarr can be obtained from the `zarr.__version__` attribute. Please also state how Zarr was installed, e.g., "installed via pip into a virtual environment", or "installed using conda". Information about other packages installed can be obtained by executing `pip freeze` (if using pip to install packages) or `conda env export` (if using conda to install packages) from the operating system command prompt. The version of the Python interpreter can be obtained by running a Python interactive session, e.g.: -```console -python -``` + ```console + python + ``` -```ansi -Python 3.12.7 | packaged by conda-forge | (main, Oct 4 2024, 15:57:01) [Clang 17.0.6 ] on darwin -``` + ```ansi + Python 3.12.7 | packaged by conda-forge | (main, Oct 4 2024, 15:57:01) [Clang 17.0.6 ] on darwin + ``` ## Enhancement proposals @@ -401,7 +401,6 @@ The Zarr library is an implementation of a file format standard defined external If an existing Zarr format version changes, or a new version of the Zarr format is released, then the Zarr library will generally require changes. It is very likely that a new Zarr format will require extensive breaking changes to the Zarr library, and so support for a new Zarr format in the Zarr library will almost certainly come in a new `major` release. When the Zarr library adds support for a new Zarr format, there may be a period of accelerated changes as developers refine newly added APIs and deprecate old APIs. In such a transitional phase breaking changes may be more frequent than usual. - ## Experimental API policy The `zarr.experimental` namespace contains features that are under active development and may change without notice. When contributing to or depending on experimental features, please keep the following in mind: diff --git a/docs/index.md b/docs/index.md index 1ec5e93c75..ee4098a8ea 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,6 @@ [Developer Chat](https://ossci.zulipchat.com/) | [Zarr specifications](https://zarr-specs.readthedocs.io) - Zarr is a powerful library for storage of n-dimensional arrays, supporting chunking, compression, and various backends, making it a versatile choice for scientific and large-scale data. @@ -37,22 +36,20 @@ conda install -c conda-forge zarr
-- [:material-clock-fast:{ .lg .middle } __Quick start__](quick-start.md) +- [:material-clock-fast:{ .lg .middle } __Quick start__](quick-start.md) --- New to Zarr? Check out the quick start guide. It contains a brief introduction to Zarr's main concepts and links to additional tutorials. - -- [:material-book-open:{ .lg .middle } __User guide__](user-guide/index.md) +- [:material-book-open:{ .lg .middle } __User guide__](user-guide/index.md) --- A detailed guide for how to use Zarr-Python. - -- [:material-api:{ .lg .middle } __API Reference__](api/zarr/index.md) +- [:material-api:{ .lg .middle } __API Reference__](api/zarr/index.md) --- @@ -61,8 +58,7 @@ conda install -c conda-forge zarr which parameters can be used. It assumes that you have an understanding of the key concepts. - -- [:material-account-group:{ .lg .middle } __Contributor's Guide__](contributing.md) +- [:material-account-group:{ .lg .middle } __Contributor's Guide__](contributing.md) --- @@ -72,7 +68,6 @@ conda install -c conda-forge zarr
- ## Project Status More information about the Zarr format can be found on the [main website](https://zarr.dev). @@ -80,6 +75,7 @@ More information about the Zarr format can be found on the [main website](https: If you are using Zarr-Python, we would [love to hear about it](https://github.com/zarr-developers/community/issues/19). ### Funding and Support + The project is fiscally sponsored by [NumFOCUS](https://numfocus.org/), a US 501(c)(3) public charity, and development has been supported by the [MRC Centre for Genomics and Global Health](https://github.com/cggh/) diff --git a/docs/quick-start.md b/docs/quick-start.md index 29232d2a04..17cb1c599a 100644 --- a/docs/quick-start.md +++ b/docs/quick-start.md @@ -50,7 +50,6 @@ written to a `LocalStore` in the `data/example-1.zarr` directory. Zarr supports data compression and filters. For example, to use Blosc compression: - ```python exec="true" session="quickstart" source="above" result="ansi" # Create a 2D Zarr array with Blosc compression @@ -73,7 +72,6 @@ print(z.info) This compresses the data using the Blosc codec with shuffle enabled for better compression. - ## Hierarchical Groups Zarr allows you to create hierarchical groups, similar to directories: diff --git a/docs/release-notes.md b/docs/release-notes.md index e85137f2b6..7a5b12f59f 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -1,5 +1,7 @@ # Release notes + + ## 3.3.0 (2026-07-15) diff --git a/docs/user-guide/attributes.md b/docs/user-guide/attributes.md index 96a5d67584..8c11c853f1 100644 --- a/docs/user-guide/attributes.md +++ b/docs/user-guide/attributes.md @@ -20,6 +20,7 @@ print('foo' in root.attrs) ```python exec="true" session="attributes" source="above" result="ansi" print(root.attrs['foo']) ``` + ```python exec="true" session="attributes" source="above" result="ansi" print(sorted(z.attrs)) ``` diff --git a/docs/user-guide/consolidated_metadata.md b/docs/user-guide/consolidated_metadata.md index 7030ed6cbc..9cb4d87c89 100644 --- a/docs/user-guide/consolidated_metadata.md +++ b/docs/user-guide/consolidated_metadata.md @@ -119,7 +119,6 @@ removed, or modified, consolidated metadata may not be desirable. `use_consolidated=False` to [`zarr.open_group`][] to always read child metadata directly from the store. - ## Stores Without Support for Consolidated Metadata Some stores may want to opt out of the consolidated metadata mechanism. This diff --git a/docs/user-guide/data_types.md b/docs/user-guide/data_types.md index 642a09d714..ac966f885f 100644 --- a/docs/user-guide/data_types.md +++ b/docs/user-guide/data_types.md @@ -194,9 +194,11 @@ Python supports nearly all of the data types in NumPy. If you need a data type t here, it's possible to create it yourself: see [Adding new data types](#adding-new-data-types). #### Boolean + - [Boolean][zarr.dtype.Bool] #### Integral + - [Signed 8-bit integer][zarr.dtype.Int8] - [Signed 16-bit integer][zarr.dtype.Int16] - [Signed 32-bit integer][zarr.dtype.Int32] @@ -207,6 +209,7 @@ here, it's possible to create it yourself: see [Adding new data types](#adding-n - [Unsigned 64-bit integer][zarr.dtype.UInt64] #### Floating-point + - [16-bit floating-point][zarr.dtype.Float16] - [32-bit floating-point][zarr.dtype.Float32] - [64-bit floating-point][zarr.dtype.Float64] @@ -214,19 +217,23 @@ here, it's possible to create it yourself: see [Adding new data types](#adding-n - [128-bit complex floating-point][zarr.dtype.Complex128] #### String + - [Fixed-length UTF-32 string][zarr.dtype.FixedLengthUTF32] - [Variable-length UTF-8 string][zarr.dtype.VariableLengthUTF8] #### Bytes + - [Fixed-length null-terminated bytes][zarr.dtype.NullTerminatedBytes] - [Fixed-length raw bytes][zarr.dtype.RawBytes] - [Variable-length bytes][zarr.dtype.VariableLengthBytes] #### Temporal + - [DateTime64][zarr.dtype.DateTime64] - [TimeDelta64][zarr.dtype.TimeDelta64] #### Struct-like + - [Structured][zarr.dtype.Structured] !!! note "Zarr V3 Structured Data Types" diff --git a/docs/user-guide/experimental.md b/docs/user-guide/experimental.md index a70ee923fe..e14146610c 100644 --- a/docs/user-guide/experimental.md +++ b/docs/user-guide/experimental.md @@ -168,7 +168,6 @@ print(f"Speedup is {speedup}") Cache effectiveness is particularly pronounced with repeated access to the same data chunks. - ### Cache Configuration The CacheStore can be configured with several parameters: diff --git a/docs/user-guide/extending.md b/docs/user-guide/extending.md index 8602ec8c3b..f852f9105e 100644 --- a/docs/user-guide/extending.md +++ b/docs/user-guide/extending.md @@ -14,6 +14,7 @@ in the following ways: [numcodecs.registry.register_codec](https://numcodecs.readthedocs.io/en/stable/registry.html#numcodecs.registry.register_codec). There are three types of codecs in Zarr: + - array-to-array - array-to-bytes - bytes-to-bytes diff --git a/docs/user-guide/groups.md b/docs/user-guide/groups.md index a2b40dfe8a..337ad39554 100644 --- a/docs/user-guide/groups.md +++ b/docs/user-guide/groups.md @@ -174,4 +174,3 @@ Groups also have the [`zarr.Group.tree`][] method, e.g.: ```python exec="true" session="groups" source="above" result="ansi" print(root.tree()) ``` - diff --git a/docs/user-guide/performance.md b/docs/user-guide/performance.md index 818386578c..52c1cf0d71 100644 --- a/docs/user-guide/performance.md +++ b/docs/user-guide/performance.md @@ -120,7 +120,6 @@ The order of chunks **within each shard** can be changed via the `subchunk_write By default [`morton`](https://en.wikipedia.org/wiki/Z-order_curve) order provides good spatial locality. [`lexicographic` (i.e., row-major)](https://en.wikipedia.org/wiki/Row-_and_column-major_order), for example, may be better suited to "batched" workflows where some form of sequential reading through a fixed number of outer dimensions is desired, and `colexicographic` is its reverse. `unordered` makes no guarantee about the order in which subchunks are laid out within a shard. - ### Empty chunks It is possible to configure how Zarr handles the storage of chunks that are "empty" @@ -221,11 +220,13 @@ zarr.config.set({'async.concurrency': 128}) ``` Higher concurrency values can improve throughput when: + - Working with remote storage (e.g., S3, GCS) where network latency is high - Reading/writing many small chunks in parallel - The storage backend can handle many concurrent requests Lower concurrency values may be beneficial when: + - Working with local storage with limited I/O bandwidth - Memory is constrained (each concurrent operation requires buffer space) - Using Zarr within a parallel computing framework (see below) @@ -258,7 +259,7 @@ concurrently. **Important**: When using many Dask threads, you may need to reduce both Zarr's `async.concurrency` and `threading.max_workers` settings to avoid creating too many concurrent operations. The total number of concurrent I/O operations can be roughly estimated as: -``` +```text total_concurrency ≈ dask_threads × zarr_async_concurrency ``` @@ -298,6 +299,7 @@ You may need to experiment with different values to find the optimal balance for Zarr arrays are designed to be thread-safe for concurrent reads and writes from multiple threads within the same process. However, proper synchronization is required when writing to overlapping regions from multiple threads. For multi-process parallelism, Zarr provides safe concurrent writes as long as: + - Different processes write to different chunks - The storage backend supports atomic writes (most do) diff --git a/docs/user-guide/storage.md b/docs/user-guide/storage.md index d32fe217c9..7e0154b2a0 100644 --- a/docs/user-guide/storage.md +++ b/docs/user-guide/storage.md @@ -41,17 +41,21 @@ group = zarr.create_group(store=data) print(group) ``` + [](){#user-guide-store-like} + ### StoreLike `StoreLike` values can be: - a `Path` or string indicating a location on the local file system. This will create a [local store](#local-store): + ```python exec="true" session="storage" source="above" result="ansi" group = zarr.open_group(store='data/foo/bar') print(group) ``` + ```python exec="true" session="storage" source="above" result="ansi" from pathlib import Path group = zarr.open_group(store=Path('data/foo/bar')) @@ -59,6 +63,7 @@ print(group) ``` - an FSSpec URI string, indicating a [remote store](#remote-store) location: + ```python exec="true" session="storage" source="above" result="ansi" # Note: requires s3fs to be installed group = zarr.open_group( @@ -70,10 +75,12 @@ print(group) ``` - an empty dictionary or None, which will create a new [memory store](#memory-store): + ```python exec="true" session="storage" source="above" result="ansi" group = zarr.create_group(store={}) print(group) ``` + ```python exec="true" session="storage" source="above" result="ansi" group = zarr.create_group(store=None) print(group) @@ -152,7 +159,6 @@ print(store) When using an S3-compatible service other than AWS, pass the service endpoint to the filesystem via `client_kwargs={'endpoint_url': 'https://...'}`. - ### Memory Store The [`zarr.storage.MemoryStore`][] stores Zarr data (metadata and chunks) in an diff --git a/lychee.toml b/lychee.toml new file mode 100644 index 0000000000..38b2b8ab7a --- /dev/null +++ b/lychee.toml @@ -0,0 +1,20 @@ +# Configuration for the lychee link checker (https://lychee.cli.rs/). +# Auto-discovered as ./lychee.toml by the lychee GitHub Action. + +# Files lychee should not scan for links. +exclude_path = [ + # mkdocs-material theme overrides: hrefs are Jinja expressions like + # `{{ '../' ~ base_url }}`, not real URLs, so lychee cannot resolve them. + "docs/overrides", + # Design notes: working records that point at transient artifacts (commits, + # fork branches, compare URLs) which are expected to disappear over time. + "design", +] + +# URL patterns to ignore (regex, matched against the full URL). +exclude = [ + # Local docs preview server shown in the contributing guide ("hatch run serve"), + # documentation of a command rather than a reachable link. + '^https?://0\.0\.0\.0', + '^https?://(localhost|127\.0\.0\.1)(:\d+)?', +] diff --git a/src/zarr/core/metadata/v2.py b/src/zarr/core/metadata/v2.py index ac32521239..91515d87b9 100644 --- a/src/zarr/core/metadata/v2.py +++ b/src/zarr/core/metadata/v2.py @@ -120,7 +120,7 @@ def ndim(self) -> int: def chunk_grid(self) -> ChunkGrid: """Backwards-compatible chunk grid property. - .. deprecated:: + !!! warning "Deprecated" Access the chunk grid via the array layer instead. This property will be removed in a future release. """ diff --git a/src/zarr/testing/store.py b/src/zarr/testing/store.py index 80a0996ca6..46287ccffb 100644 --- a/src/zarr/testing/store.py +++ b/src/zarr/testing/store.py @@ -40,21 +40,21 @@ class StoreTests[S: Store, B: Buffer]: @staticmethod def _require_get_sync(store: S) -> SupportsGetSync: - """Skip unless *store* implements :class:`SupportsGetSync`.""" + """Skip unless *store* implements [`SupportsGetSync`][zarr.abc.store.SupportsGetSync].""" if not isinstance(store, SupportsGetSync): pytest.skip("store does not implement SupportsGetSync") return store # type: ignore[unreachable] @staticmethod def _require_set_sync(store: S) -> SupportsSetSync: - """Skip unless *store* implements :class:`SupportsSetSync`.""" + """Skip unless *store* implements [`SupportsSetSync`][zarr.abc.store.SupportsSetSync].""" if not isinstance(store, SupportsSetSync): pytest.skip("store does not implement SupportsSetSync") return store # type: ignore[unreachable] @staticmethod def _require_delete_sync(store: S) -> SupportsDeleteSync: - """Skip unless *store* implements :class:`SupportsDeleteSync`.""" + """Skip unless *store* implements [`SupportsDeleteSync`][zarr.abc.store.SupportsDeleteSync].""" if not isinstance(store, SupportsDeleteSync): pytest.skip("store does not implement SupportsDeleteSync") return store # type: ignore[unreachable] From 7f8834ab14d607d1ec03209d9509a153fe81b173 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Thu, 16 Jul 2026 13:42:21 -0400 Subject: [PATCH 06/16] chore: cleanup indentation and guidance for release (#4153) * chore: cleanup indentation and guidance for release * Make explanation a comment --- .github/ISSUE_TEMPLATE/release-checklist.md | 7 +- .github/PULL_REQUEST_TEMPLATE.md | 2 +- .github/workflows/prepare_release.yml | 74 ------ ci/lint_docs.py | 67 ++++- docs/contributing.md | 21 +- docs/release-notes.md | 258 ++++++++++---------- docs/user-guide/data_types.md | 20 +- docs/user-guide/v3_migration.md | 73 +++--- 8 files changed, 251 insertions(+), 271 deletions(-) delete mode 100644 .github/workflows/prepare_release.yml diff --git a/.github/ISSUE_TEMPLATE/release-checklist.md b/.github/ISSUE_TEMPLATE/release-checklist.md index 5e073cb59f..309c76b4dc 100644 --- a/.github/ISSUE_TEMPLATE/release-checklist.md +++ b/.github/ISSUE_TEMPLATE/release-checklist.md @@ -18,8 +18,8 @@ assignees: '' - [ ] Check [SPEC 0](https://scientific-python.org/specs/spec-0000/#support-window) to see if the minimum supported version of Python or NumPy needs bumping. - [ ] Verify that the latest CI workflows on `main` are passing: [Tests](https://github.com/zarr-developers/zarr-python/actions/workflows/test.yml), [GPU Tests](https://github.com/zarr-developers/zarr-python/actions/workflows/gpu_test.yml), [Hypothesis](https://github.com/zarr-developers/zarr-python/actions/workflows/hypothesis.yaml), [Docs](https://github.com/zarr-developers/zarr-python/actions/workflows/docs.yml), [Lint](https://github.com/zarr-developers/zarr-python/actions/workflows/lint.yml), [Wheels](https://github.com/zarr-developers/zarr-python/actions/workflows/releases.yml). -- [ ] Run the ["Prepare release" workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) with the target version. This will build the changelog and open a release PR with the `run-downstream` label. -- [ ] Verify that the [downstream tests](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) (triggered automatically by the `run-downstream` label) pass on the release PR. +- [ ] Run the [downstream tests](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) against `main`: go to the workflow page, click "Run workflow", and select the `main` branch. Verify that the Xarray and numcodecs integration tests pass. +- [ ] Open a release PR with the changelog entries for the upcoming release, generated with `uv run --only-group release towncrier build --version x.y.z`. - [ ] Review the release PR and verify the changelog in `docs/release-notes.md` looks correct. - [ ] Merge the release PR. @@ -45,7 +45,8 @@ In rare cases (e.g. patch releases for an older minor version), you may need to - Create the release branch from the appropriate tag if it doesn't already exist. - Cherry-pick or backport the necessary commits onto the branch. -- Run `towncrier build --version x.y.z` and commit the result to the release branch instead of `main`. +- Run `towncrier build --version x.y.z` and open the release PR against the release branch instead of `main`. +- Run the downstream tests against the release branch instead of `main`. - When drafting the GitHub Release, set the target to the release branch instead of `main`. - After the release, ensure any relevant changelog updates are also reflected on `main`. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1d12aa02eb..47adb4b19e 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -15,7 +15,7 @@ - [ ] I am a human, these are my changes, and I have reviewed and understood every change and can explain why each is correct. -AI coding assistance is welcome, but a human must be the author and is responsible for the contents of the PR. The description and any review responses must be in your own words. Please read [AI-assisted contributions](https://zarr.readthedocs.io/en/stable/contributing/#ai-assisted-contributions) before opening. + ## TODO diff --git a/.github/workflows/prepare_release.yml b/.github/workflows/prepare_release.yml deleted file mode 100644 index d558779bf2..0000000000 --- a/.github/workflows/prepare_release.yml +++ /dev/null @@ -1,74 +0,0 @@ -name: Prepare release notes - -on: - workflow_dispatch: - inputs: - version: - description: 'Release version notes (e.g. 3.2.0)' - required: true - type: string - target_branch: - description: 'Branch to target' - required: false - default: 'main' - type: string - -permissions: - contents: write - pull-requests: write - -jobs: - prepare: - name: Build changelog and open PR - runs-on: ubuntu-latest - steps: - - name: Validate inputs - run: | - if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+([-\.][a-zA-Z0-9]+)*$ ]]; then - echo "::error::Invalid version format: '$VERSION'" - exit 1 - fi - if [[ ! "$TARGET_BRANCH" =~ ^[a-zA-Z0-9._/-]+$ ]]; then - echo "::error::Invalid branch name: '$TARGET_BRANCH'" - exit 1 - fi - env: - VERSION: ${{ inputs.version }} - TARGET_BRANCH: ${{ inputs.target_branch }} - - - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 - with: - ref: ${{ inputs.target_branch }} - fetch-depth: 0 - persist-credentials: false - - - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 - - - name: Build changelog - # Use the pinned towncrier from the `release` dependency group (single - # source of truth) rather than an unpinned standalone install. - run: uv run --only-group release towncrier build --version "$VERSION" --yes - env: - VERSION: ${{ inputs.version }} - - - name: Create pull request - uses: peter-evans/create-pull-request@5f6978faf089d4d20b00c7766989d076bb2fc7f1 # v8.1.1 - with: - branch: release/v${{ inputs.version }} - base: ${{ inputs.target_branch }} - title: "Release v${{ inputs.version }}" - body: | - Automated release preparation for v${{ inputs.version }}. - - This PR was generated by the "Prepare release" workflow. It includes: - - Rendered changelog via `towncrier build --version ${{ inputs.version }}` - - Removal of consumed changelog fragments from `changes/` - - ## Checklist - - - [ ] Review the rendered changelog in `docs/release-notes.md` - - [ ] Downstream tests pass (see [downstream workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml)) - - [ ] Merge this PR, then [draft a GitHub Release](https://github.com/zarr-developers/zarr-python/releases/new) targeting `${{ inputs.target_branch }}` with tag `v${{ inputs.version }}` - commit-message: "chore: build changelog for v${{ inputs.version }}" - labels: run-downstream - delete-branch: true diff --git a/ci/lint_docs.py b/ci/lint_docs.py index ff7b56649b..a847e8aa9d 100644 --- a/ci/lint_docs.py +++ b/ci/lint_docs.py @@ -13,7 +13,7 @@ fast, source-level check that needs no docs build. Checks fall into two groups -- RST markup that silently fails under MkDocs/mkdocstrings, -and a Markdown structural problem that renders as valid-but-wrong HTML (so `mkdocs build` +and Markdown structural problems that render as valid-but-wrong HTML (so `mkdocs build` emits no warning): sphinx-role :class:`X`, :func:`X`, :py:meth:`X` -> [`X`][zarr.X] @@ -21,6 +21,8 @@ rst-field :param x:, :returns:, :rtype: -> numpydoc Parameters/Returns/Raises rst-link `text `_ -> [text](https://example) list-break unindented code fence between list items -> indent the fence under its item + list-indent continuation block indented < 4 spaces -> indent it 4 spaces + list-blank list item directly after indented block -> blank line before the item The ``list-break`` check catches a fenced code block at column 0 placed *between* two list items: because the fence is not indented into the preceding item, Markdown ends the list at @@ -29,6 +31,15 @@ notices this for sequentially-numbered ordered lists; lazily-numbered (1., 1.) and unordered lists slip past it, so this structural check covers the gap. +The ``list-indent`` and ``list-blank`` checks catch the two halves of Python-Markdown's +strict list-continuation rules, which differ from CommonMark. A blank-line-separated +block (paragraph, nested list, table) belongs to a list item only when indented at least +4 spaces; at the 2-space indent other renderers accept, Python-Markdown ends the list and +the block escapes to the top level (``list-indent``). And a new list item can not start +directly after an indented continuation block: without a blank line first, the ``- `` line +is lazily absorbed into the preceding paragraph as literal text (``list-blank``). Both +produced silently-broken changelog rendering in ``docs/release-notes.md``. + Usage: python ci/lint_docs.py [PATH ...] @@ -87,14 +98,16 @@ class Check(NamedTuple): fix: str -# ``list-break`` carries no pattern -- it is detected structurally in find_list_breaking_fences, -# not by scanning a single line -- but it appears here so it shares the remediation help. +# The ``list-*`` checks carry no pattern -- they are detected structurally, not by scanning +# a single line -- but they appear here so they share the remediation help. CHECKS = ( Check("sphinx-role", SPHINX_ROLE, ":class:`X`", "[`X`][zarr.X]"), Check("rst-directive", RST_DIRECTIVE, ".. note::", "MkDocs admonition (!!! note)"), Check("rst-field", RST_FIELD, ":param x:", "numpydoc Parameters/Returns/Raises section"), Check("rst-link", RST_LINK, "`text `_", "[text](url)"), Check("list-break", None, "fence between items", "indent the fence 4 spaces to nest it"), + Check("list-indent", None, "2-space continuation", "indent the block 4 spaces under its item"), + Check("list-blank", None, "item after indented block", "add a blank line before the item"), ) @@ -211,9 +224,49 @@ def splits_a_list(open_i: int, close_i: int) -> bool: ] +def find_list_continuation_issues( + lines: list[str], in_code: set[int] +) -> list[tuple[int, str, str]]: + """Return ``(lineno, category, snippet)`` for list continuations Python-Markdown will + mis-render (see the module docstring): + + - ``list-indent``: a blank-line-separated block inside a list item indented 1-3 + spaces. Python-Markdown requires 4; at less, the block escapes the list. + - ``list-blank``: a top-level list item directly after a line indented 4+ spaces. + Without a blank line in between, the item is absorbed into the preceding paragraph + as literal ``- `` text. + + Lazy continuations (an indented line with no blank line before it) are valid at any + indent and are not flagged. Fenced-code lines are opaque: never flagged themselves, + but they keep the item scope open and their indent feeds the ``list-blank`` check so + an item directly after an indented fence is still caught.""" + findings: list[tuple[int, str, str]] = [] + in_item = False # inside a top-level list item's scope + prev_blank = True + prev_indent = 0 + for i, line in enumerate(lines): + stripped = line.strip() + if not stripped: + prev_blank = True + continue + indent = len(line) - len(line.lstrip(" ")) + if i not in in_code: + if indent == 0: + is_item = bool(LIST_ITEM.match(line)) + if is_item and in_item and not prev_blank and prev_indent >= 4: + findings.append((i + 1, "list-blank", line)) + in_item = is_item + elif in_item and prev_blank and indent < 4: + findings.append((i + 1, "list-indent", line)) + prev_blank = False + prev_indent = indent + return findings + + def lint_markdown(path: Path) -> list[Finding]: """Scan a Markdown file: RST residue in prose (skipping fenced code blocks), plus - fenced code blocks that break a list (see find_list_breaking_fences).""" + list-structure problems (see find_list_breaking_fences and + find_list_continuation_issues).""" lines = path.read_text(encoding="utf-8").splitlines() blocks = fenced_blocks(lines) in_code = {i for fence in blocks for i in range(fence.open, fence.close + 1)} @@ -228,7 +281,11 @@ def lint_markdown(path: Path) -> list[Finding]: Finding(path, lineno, "list-break", snippet) for lineno, snippet in find_list_breaking_fences(lines, blocks) ] - return prose + breaks + continuations = [ + Finding(path, lineno, category, snippet) + for lineno, category, snippet in find_list_continuation_issues(lines, in_code) + ] + return prose + breaks + continuations def iter_files(paths: tuple[Path, ...]) -> list[Path]: diff --git a/docs/contributing.md b/docs/contributing.md index df46f381de..aeb88e6ce1 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -354,18 +354,7 @@ This policy exists to lower the cost of routine work and to help newer core deve ### Release procedure -Open an issue on GitHub announcing the release using the release checklist template: -[https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md](https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md). The release checklist includes all steps necessary for the release. - -#### Preparing a release - -Releases are prepared using the ["Prepare release notes"](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) workflow. To run it: - -1. Go to the [workflow page](https://github.com/zarr-developers/zarr-python/actions/workflows/prepare_release.yml) and click "Run workflow". -2. Enter the release version (e.g. `3.2.0`) and the target branch (defaults to `main`). -3. The workflow will run `towncrier build` to render the changelog, remove consumed fragments from `changes/`, and open a pull request on the `release/v` branch. -4. The release PR is automatically labeled `run-downstream`, which triggers the [downstream test workflow](https://github.com/zarr-developers/zarr-python/actions/workflows/downstream.yml) to run Xarray and numcodecs integration tests against the release branch. -5. Review the rendered changelog in `docs/release-notes.md` and verify downstream tests pass before merging. +To give the release visibility and a single place to track progress, open an issue on GitHub announcing the release using the [release checklist template](https://github.com/zarr-developers/zarr-python/issues/new?template=release-checklist.md). The release checklist includes all steps necessary for the release. ## Compatibility and versioning policies @@ -377,17 +366,17 @@ Releases are classified by the library changes contained in that release. This c * **major** releases (for example, `2.18.0` -> `3.0.0`) are for changes that will require extensive adaptation efforts from many users and downstream projects. For example, breaking changes to widely-used user-facing APIs should only be applied in a major release. - Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. + Users and downstream projects should carefully consider the impact of a major release before adopting it. In advance of a major release, developers should communicate the scope of the upcoming changes, and help users prepare for them. * **minor** releases (for example, `3.0.0` -> `3.1.0`) are for changes that do not require significant effort from most users or downstream projects to respond to. API changes are possible in minor releases if the burden on users imposed by those changes is sufficiently small. - For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. + For example, a recently released API may need fixes or refinements that are breaking, but low impact due to the recency of the feature. Such API changes are permitted in a minor release. - Minor releases are safe for most users and downstream projects to adopt. + Minor releases are safe for most users and downstream projects to adopt. * **patch** releases (for example, `3.1.0` -> `3.1.1`) are for changes that contain no breaking or behavior changes for downstream projects or users. Examples of changes suitable for a patch release are bugfixes and documentation improvements. - Users should always feel safe upgrading to the latest patch release. + Users should always feel safe upgrading to the latest patch release. Note that this versioning scheme is not consistent with [Semantic Versioning](https://semver.org/). Contrary to SemVer, the Zarr library may release breaking changes in `minor` releases, or even `patch` releases under exceptional circumstances. But we should strive to avoid doing so. diff --git a/docs/release-notes.md b/docs/release-notes.md index 7a5b12f59f..3fd8a5f360 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -51,32 +51,34 @@ `ClientSession` was left open until garbage collection, producing `"Unclosed client session"` `ResourceWarning`s from aiohttp. - The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when - `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` - when a sync→async conversion was performed). When `_owns_fs` is ``True``, - `store.close()` calls the new `_close_fs()` helper, which invokes - `fs.set_session()` and closes the returned client. Callers who supply their own - filesystem instance to `FsspecStore()` directly remain responsible for its - lifecycle; `_owns_fs` is ``False`` for those stores. - - **Scope note**: This fix closes the S3 client session that is active at the time - `store.close()` is called. Some S3-backed filesystem implementations (e.g. - s3fs with ``cache_regions=True``) may internally refresh and replace their - client during I/O operations, abandoning prior sessions before ``store.close()`` - is invoked. Those intermediate sessions are outside the scope of this fix and - are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) + The fix introduces `FsspecStore._owns_fs`, a boolean that is ``True`` only when + `FsspecStore` itself created the filesystem (via `from_url` or `from_mapper` + when a sync→async conversion was performed). When `_owns_fs` is ``True``, + `store.close()` calls the new `_close_fs()` helper, which invokes + `fs.set_session()` and closes the returned client. Callers who supply their own + filesystem instance to `FsspecStore()` directly remain responsible for its + lifecycle; `_owns_fs` is ``False`` for those stores. + + **Scope note**: This fix closes the S3 client session that is active at the time + `store.close()` is called. Some S3-backed filesystem implementations (e.g. + s3fs with ``cache_regions=True``) may internally refresh and replace their + client during I/O operations, abandoning prior sessions before ``store.close()`` + is invoked. Those intermediate sessions are outside the scope of this fix and + are an issue in the upstream filesystem library. ([#4003](https://github.com/zarr-developers/zarr-python/issues/4003)) + - Fixed an invalid `zarr.create_array` example in the quick-start documentation (it passed an unsupported `mode` argument) and made the cloud-storage example execute against a mock S3 backend in CI. Added a test ensuring every Python code block in the documentation is either executed or explicitly opted out with a documented reason, so an invalid example can no longer go untested. ([#4016](https://github.com/zarr-developers/zarr-python/issues/4016)) - Fixed `ObjectStore.list_dir` for object-store listings that include a directory-marker object matching the requested non-root prefix. ([#4032](https://github.com/zarr-developers/zarr-python/issues/4032)) - Prevents mutation of the attributes dict provided by the user by copying them instead of keeping the reference ([#4059](https://github.com/zarr-developers/zarr-python/issues/4059)) - Fixed several storage and codec bugs: - - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. - - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). - - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). - - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). - - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + - Reading a value with a `SuffixByteRequest` larger than the value now correctly returns the whole value (matching HTTP `bytes=-N` suffix-range semantics), instead of silently returning incorrect data for `MemoryStore`. + - `LoggingStore.get_partial_values` and `FsspecStore.get_partial_values` no longer return empty results when `key_ranges` is passed as a one-shot iterable (e.g. a generator). + - `Store.getsize_prefix` no longer over-counts sibling keys that merely share a string prefix (e.g. `getsize_prefix("foo")` no longer includes keys under `foobar/`). + - `ZipStore.close()` no longer raises `AttributeError` when the store was created but never opened (including when used as a context manager without any I/O). + - `codecs_from_list` now raises a descriptive `TypeError` when a `BytesBytesCodec` immediately follows an `ArrayArrayCodec`, instead of a misleading "Required ArrayBytesCodec was not found" `ValueError`. + + ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) - ([#4074](https://github.com/zarr-developers/zarr-python/issues/4074)) - Fixed writing Fortran-ordered (F-contiguous) arrays through the variable-length string and bytes codecs and through numcodecs array-array filters such as `Delta`, `FixedScaleOffset` and `PackBits`. Chunks are now passed to numcodecs as C-contiguous arrays, so elements are no longer stored in transposed order. ([#4116](https://github.com/zarr-developers/zarr-python/issues/4116)) - Fix silent byte-order corruption for structured dtypes with the `bytes` codec: multi-byte fields are now byte-swapped to the codec's configured `endian` on write and decoded honoring it on read, so non-native-endian structured data (e.g. big-endian fields, as produced by virtual references to external data) round-trips correctly. ([#4141](https://github.com/zarr-developers/zarr-python/issues/4141)) @@ -91,23 +93,24 @@ use only the public API, eliminating all non-public imports, illustrating what users should do. - To better support this, the following types and functions were made available - from public modules: - - | Type/Function | Non-public module | Public module | - | ------------------------- | ------------------------ | ------------- | - | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | - | `JSON` | `zarr.core.common` | `zarr.types` | - | `ZarrFormat` | `zarr.core.common` | `zarr.types` | - | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | - | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | - | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | - | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | - - `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from - `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or - `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining - types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) + To better support this, the following types and functions were made available + from public modules: + + | Type/Function | Non-public module | Public module | + | ------------------------- | ------------------------ | ------------- | + | `DataTypeValidationError` | `zarr.core.dtype.common` | `zarr.errors` | + | `JSON` | `zarr.core.common` | `zarr.types` | + | `ZarrFormat` | `zarr.core.common` | `zarr.types` | + | `DTypeConfig_V2` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeJSON` | `zarr.core.dtype.common` | `zarr.types` | + | `DTypeSpec_V2` | `zarr.core.dtype.common` | `zarr.dtype` | + | `check_dtype_spec_v2` | `zarr.core.dtype.common` | `zarr.dtype` | + + `DataTypeValidationError` was *moved* to `zarr.errors`. Importing it from + `zarr.core.dtype.common` (its original location), `zarr.core.dtype`, or + `zarr.dtype` still works but now raises a `ZarrDeprecationWarning`. The remaining + types and functions are simply re-exported from the listed public module. ([#4052](https://github.com/zarr-developers/zarr-python/issues/4052)) + - Document a self-merge policy in the contributor guide, describing when a core developer may merge their own pull request without a second reviewer and which changes warrant more caution. ([#4053](https://github.com/zarr-developers/zarr-python/issues/4053)) - Fixed many documentation errors found in a full review of the user guide, including prose contradicted by rendered example output on the performance page, invisible @@ -126,11 +129,12 @@ access, and will be removed in a future release. ``BloscCodec.cname`` and ``BloscCodec.shuffle`` are now plain strings rather than enum members. - Additional renames in ``zarr.codecs.blosc`` from the same change: the type - aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and - ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` - (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` - now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) + Additional renames in ``zarr.codecs.blosc`` from the same change: the type + aliases ``Shuffle`` and ``CName`` are now ``BloscShuffleLiteral`` and + ``BloscCnameLiteral``, the constant ``SHUFFLE`` is now ``BLOSC_SHUFFLE`` + (with a new ``BLOSC_CNAME`` alongside it), and ``BloscShuffle.from_int`` + now returns a literal string rather than an enum member. ([#3963](https://github.com/zarr-developers/zarr-python/issues/3963)) + - The ``Endian`` (``zarr.codecs.bytes.Endian``) and ``ShardingCodecIndexLocation`` (``zarr.codecs.ShardingCodecIndexLocation``) enums are now deprecated. Pass the equivalent literal string instead (e.g. ``"little"`` / ``"big"``, ``"start"`` / @@ -139,20 +143,21 @@ and ``ShardingCodec.index_location`` are now plain strings rather than enum members. - Two follow-on changes from this deprecation: + Two follow-on changes from this deprecation: - - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or - ``"big"``) rather than an ``Endian`` member. Subclasses overriding this - property should update their return type. - - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was - removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; - external callers that imported ``default_system_endian`` should use - ``sys.byteorder`` directly. + - ``NDBuffer.byteorder`` now returns a literal string (``"little"`` or + ``"big"``) rather than an ``Endian`` member. Subclasses overriding this + property should update their return type. + - The module-level binding ``zarr.codecs.bytes.default_system_endian`` was + removed. ``BytesCodec()`` continues to default to ``sys.byteorder``; + external callers that imported ``default_system_endian`` should use + ``sys.byteorder`` directly. - Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` - was made private as part of this change. + Additionally, the module-level function ``zarr.codecs.sharding.parse_index_location`` + was made private as part of this change. + + ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) - ([#3968](https://github.com/zarr-developers/zarr-python/issues/3968)) - Removed the NumPy 1.x implementation of the `VariableLengthUTF8` data type because NumPy 1.x is no longer supported under [SPEC0](https://scientific-python.org/specs/spec-0000/). ([#3973](https://github.com/zarr-developers/zarr-python/issues/3973)) ### Misc @@ -191,19 +196,19 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Add support for rectilinear (variable-sized) chunk grids. This feature is experimental and must be explicitly enabled via `zarr.config.set({'array.rectilinear_chunks': True})`. - Rectilinear chunks can be used through: + Rectilinear chunks can be used through: - - **Creating arrays**: Pass nested sequences (e.g., `[[10, 20, 30], [50, 50]]`) to `chunks` - in `zarr.create_array`, `zarr.from_array`, `zarr.zeros`, `zarr.ones`, `zarr.full`, - `zarr.open`, and related functions, or to `chunk_shape` in `zarr.create`. - - **Opening existing arrays**: Arrays stored with the `rectilinear` chunk grid are read - transparently via `zarr.open` and `zarr.open_array`. - - **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular. + - **Creating arrays**: Pass nested sequences (e.g., `[[10, 20, 30], [50, 50]]`) to `chunks` + in `zarr.create_array`, `zarr.from_array`, `zarr.zeros`, `zarr.ones`, `zarr.full`, + `zarr.open`, and related functions, or to `chunk_shape` in `zarr.create`. + - **Opening existing arrays**: Arrays stored with the `rectilinear` chunk grid are read + transparently via `zarr.open` and `zarr.open_array`. + - **Rectilinear sharding**: Shard boundaries can be rectilinear while inner chunks remain regular. - **Breaking change**: The `validate` method on `BaseCodec` and `CodecPipeline` now receives - a `ChunkGridMetadata` instance instead of a `ChunkGrid` instance for the `chunk_grid` - parameter. Third-party codecs that override `validate` and inspect the chunk grid will need to - update their type annotations. No known downstream packages were using this parameter. ([#3802](https://github.com/zarr-developers/zarr-python/issues/3802)) + **Breaking change**: The `validate` method on `BaseCodec` and `CodecPipeline` now receives + a `ChunkGridMetadata` instance instead of a `ChunkGrid` instance for the `chunk_grid` + parameter. Third-party codecs that override `validate` and inspect the chunk grid will need to + update their type annotations. No known downstream packages were using this parameter. ([#3802](https://github.com/zarr-developers/zarr-python/issues/3802)) - Add `cast_value` and `scale_offset` codecs. ([#3874](https://github.com/zarr-developers/zarr-python/issues/3874)) @@ -394,52 +399,52 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Ensure that invocations of `create_array` use consistent keyword arguments, with consistent defaults. - [`zarr.api.synchronous.create_array`][] now takes a `write_data` keyword argument - The `Group.create_array` method takes `data` and `write_data` keyword arguments. - The functions [`zarr.api.asynchronous.create`][], [`zarr.api.asynchronous.create_array`] - and the methods `Group.create_array`, `Group.array`, had the default - `fill_value` changed from `0` to the `DEFAULT_FILL_VALUE` value, which instructs Zarr to - use the default scalar value associated with the array's data type as the fill value. These are - all functions or methods for array creation that mirror, wrap or are wrapped by, another function - that already has a default `fill_value` set to `DEFAULT_FILL_VALUE`. This change is necessary - to make these functions consistent across the entire codebase, but as this changes default values, - new data might have a different fill value than expected after this change. - - For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this - change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode - strings, the default fill value might be different after this change. Users who were relying on how - Zarr interpreted `0` as a non-numeric scalar value should set their desired fill value explicitly - after this change. + [`zarr.api.synchronous.create_array`][] now takes a `write_data` keyword argument + The `Group.create_array` method takes `data` and `write_data` keyword arguments. + The functions [`zarr.api.asynchronous.create`][], [`zarr.api.asynchronous.create_array`] + and the methods `Group.create_array`, `Group.array`, had the default + `fill_value` changed from `0` to the `DEFAULT_FILL_VALUE` value, which instructs Zarr to + use the default scalar value associated with the array's data type as the fill value. These are + all functions or methods for array creation that mirror, wrap or are wrapped by, another function + that already has a default `fill_value` set to `DEFAULT_FILL_VALUE`. This change is necessary + to make these functions consistent across the entire codebase, but as this changes default values, + new data might have a different fill value than expected after this change. + + For data types where 0 is meaningful, like integers or floats, the default scalar is 0, so this + change should not be noticeable. For data types where 0 is ambiguous, like fixed-length unicode + strings, the default fill value might be different after this change. Users who were relying on how + Zarr interpreted `0` as a non-numeric scalar value should set their desired fill value explicitly + after this change. - Added public API for Buffer ABCs and implementations. - Use `zarr.buffer` to access buffer implementations, and - `zarr.abc.buffer` for the interface to implement new buffer types. + Use `zarr.buffer` to access buffer implementations, and + `zarr.abc.buffer` for the interface to implement new buffer types. - Users previously importing buffer from `zarr.core.buffer` should update their - imports to use `zarr.buffer`. As a reminder, all of `zarr.core` is - considered a private API that's not covered by zarr-python's versioning policy. ([#2871](https://github.com/zarr-developers/zarr-python/issues/2871)) + Users previously importing buffer from `zarr.core.buffer` should update their + imports to use `zarr.buffer`. As a reminder, all of `zarr.core` is + considered a private API that's not covered by zarr-python's versioning policy. ([#2871](https://github.com/zarr-developers/zarr-python/issues/2871)) - Adds zarr-specific data type classes. - This change adds a `ZDType` base class for Zarr V2 and Zarr V3 data types. Child classes are - defined for each NumPy data type. Each child class defines routines for `JSON` serialization. - New data types can be created and registered dynamically. + This change adds a `ZDType` base class for Zarr V2 and Zarr V3 data types. Child classes are + defined for each NumPy data type. Each child class defines routines for `JSON` serialization. + New data types can be created and registered dynamically. - Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays, - we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both - of these systems proved hard to extend. + Prior to this change, Zarr Python had two streams for handling data types. For Zarr V2 arrays, + we used NumPy data type identifiers. For Zarr V3 arrays, we used a fixed set of string enums. Both + of these systems proved hard to extend. - This change is largely internal, but it does change the type of the `dtype` and `data_type` - fields on the `ArrayV2Metadata` and `ArrayV3Metadata` classes. Previously, `ArrayV2Metadata.dtype` - was a NumPy `dtype` object, and `ArrayV3Metadata.data_type` was an internally-defined `enum`. - After this change, both `ArrayV2Metadata.dtype` and `ArrayV3Metadata.data_type` are instances of - `ZDType`. A NumPy data type can be generated from a `ZDType` via the `ZDType.to_native_dtype()` - method. The internally-defined Zarr V3 `enum` class is gone entirely, but the `ZDType.to_json(zarr_format=3)` - method can be used to generate either a string, or dictionary that has a string `name` field, that - represents the string value previously associated with that `enum`. + This change is largely internal, but it does change the type of the `dtype` and `data_type` + fields on the `ArrayV2Metadata` and `ArrayV3Metadata` classes. Previously, `ArrayV2Metadata.dtype` + was a NumPy `dtype` object, and `ArrayV3Metadata.data_type` was an internally-defined `enum`. + After this change, both `ArrayV2Metadata.dtype` and `ArrayV3Metadata.data_type` are instances of + `ZDType`. A NumPy data type can be generated from a `ZDType` via the `ZDType.to_native_dtype()` + method. The internally-defined Zarr V3 `enum` class is gone entirely, but the `ZDType.to_json(zarr_format=3)` + method can be used to generate either a string, or dictionary that has a string `name` field, that + represents the string value previously associated with that `enum`. - For more on this new feature, see the [documentation](user-guide/data_types.md) ([#2874](https://github.com/zarr-developers/zarr-python/issues/2874)) + For more on this new feature, see the [documentation](user-guide/data_types.md) ([#2874](https://github.com/zarr-developers/zarr-python/issues/2874)) - Added `NDBuffer.empty` method for faster ndbuffer initialization. ([#3191](https://github.com/zarr-developers/zarr-python/issues/3191)) @@ -451,10 +456,10 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Fixes a variety of issues related to string data types. - - Brings the `VariableLengthUTF8` data type Zarr V3 identifier in alignment with Zarr Python 3.0.8 - - Disallows creation of 0-length fixed-length data types - - Adds a regression test for the `VariableLengthUTF8` data type that checks against version 3.0.8 - - Allows users to request the `VariableLengthUTF8` data type with `str`, `"str"`, or `"string"`. ([#3170](https://github.com/zarr-developers/zarr-python/issues/3170)) + - Brings the `VariableLengthUTF8` data type Zarr V3 identifier in alignment with Zarr Python 3.0.8 + - Disallows creation of 0-length fixed-length data types + - Adds a regression test for the `VariableLengthUTF8` data type that checks against version 3.0.8 + - Allows users to request the `VariableLengthUTF8` data type with `str`, `"str"`, or `"string"`. ([#3170](https://github.com/zarr-developers/zarr-python/issues/3170)) - Add human readable size for No. bytes stored to `info_complete` ([#3190](https://github.com/zarr-developers/zarr-python/issues/3190)) @@ -475,26 +480,26 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Add a description on how to create a RemoteStore of a specific filesystem to the `Remote Store` section in `docs/user-guide/storage.md`. State in the docstring of `FsspecStore.from_url` that the filesystem type is inferred from the URL scheme. - It should help a user handling the case when the type of FsspecStore doesn't match the URL scheme. ([#3212](https://github.com/zarr-developers/zarr-python/issues/3212)) + It should help a user handling the case when the type of FsspecStore doesn't match the URL scheme. ([#3212](https://github.com/zarr-developers/zarr-python/issues/3212)) ### Deprecations and Removals - Removes default chunk encoding settings (filters, serializer, compressors) from the global configuration object. - This removal is justified on the basis that storing chunk encoding settings in the config required - a brittle, confusing, and inaccurate categorization of array data types, which was particularly - unsuitable after the recent addition of new data types that didn't fit naturally into the - pre-existing categories. + This removal is justified on the basis that storing chunk encoding settings in the config required + a brittle, confusing, and inaccurate categorization of array data types, which was particularly + unsuitable after the recent addition of new data types that didn't fit naturally into the + pre-existing categories. - The default chunk encoding is the same (Zstandard compression, and the required object codecs for - variable length data types), but the chunk encoding is now generated by functions that cannot be - reconfigured at runtime. Users who relied on setting the default chunk encoding via the global configuration object should - instead specify the desired chunk encoding explicitly when creating an array. + The default chunk encoding is the same (Zstandard compression, and the required object codecs for + variable length data types), but the chunk encoding is now generated by functions that cannot be + reconfigured at runtime. Users who relied on setting the default chunk encoding via the global configuration object should + instead specify the desired chunk encoding explicitly when creating an array. - This change also adds an extra validation step to the creation of Zarr V2 arrays, which ensures that - arrays with a `VariableLengthUTF8` or `VariableLengthBytes` data type cannot be created without the - correct "object codec". ([#3228](https://github.com/zarr-developers/zarr-python/issues/3228)) + This change also adds an extra validation step to the creation of Zarr V2 arrays, which ensures that + arrays with a `VariableLengthUTF8` or `VariableLengthBytes` data type cannot be created without the + correct "object codec". ([#3228](https://github.com/zarr-developers/zarr-python/issues/3228)) - Removes support for passing keyword-only arguments positionally to the following functions and methods: `save_array`, `open`, `group`, `open_group`, `create`, `get_basic_selection`, `set_basic_selection`, @@ -546,8 +551,8 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr For large arrays this will result in smaller default chunk sizes. To retain previous behaviour, explicitly set the chunk shape to the data shape. - This fix matches the existing chunking behaviour of - `zarr.save_array` and `zarr.api.asynchronous.AsyncArray.create`. ([#3103](https://github.com/zarr-developers/zarr-python/issues/3103)) + This fix matches the existing chunking behaviour of + `zarr.save_array` and `zarr.api.asynchronous.AsyncArray.create`. ([#3103](https://github.com/zarr-developers/zarr-python/issues/3103)) - When `zarr.save` has an argument `path=some/path/` and multiple arrays in `args`, the path resulted in `some/path/some/path` due to using the `path` argument twice while building the array path. This is now fixed. ([#3127](https://github.com/zarr-developers/zarr-python/issues/3127)) @@ -556,12 +561,12 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - Suppress `FileNotFoundError` when deleting non-existent keys in the `obstore` adapter. - When writing empty chunks (i.e. chunks where all values are equal to the array's fill value) to a zarr array, zarr - will delete those chunks from the underlying store. For zarr arrays backed by the `obstore` adapter, this will potentially - raise a `FileNotFoundError` if the chunk doesn't already exist. - Since whether or not a delete of a non-existing object raises an error depends on the behavior of the underlying store, - suppressing the error in all cases results in consistent behavior across stores, and is also what `zarr` seems to expect - from the store. ([#3140](https://github.com/zarr-developers/zarr-python/issues/3140)) + When writing empty chunks (i.e. chunks where all values are equal to the array's fill value) to a zarr array, zarr + will delete those chunks from the underlying store. For zarr arrays backed by the `obstore` adapter, this will potentially + raise a `FileNotFoundError` if the chunk doesn't already exist. + Since whether or not a delete of a non-existing object raises an error depends on the behavior of the underlying store, + suppressing the error in all cases results in consistent behavior across stores, and is also what `zarr` seems to expect + from the store. ([#3140](https://github.com/zarr-developers/zarr-python/issues/3140)) - Trying to open a StorePath/Array with `mode='r'` when the store is not read-only creates a read-only copy of the store. ([#3156](https://github.com/zarr-developers/zarr-python/issues/3156)) @@ -583,8 +588,9 @@ a `ManagedMemoryStore` is used. ([#3944](https://github.com/zarr-developers/zarr - It is now possible to specify no compressor when creating a zarr format 2 array. This can be done by passing `compressor=None` to the various array creation routines. - The default behaviour of automatically choosing a suitable default compressor remains if the compressor argument is not given. - To reproduce the behaviour in previous zarr-python versions when `compressor=None` was passed, pass `compressor='auto'` instead. ([#3039](https://github.com/zarr-developers/zarr-python/issues/3039)) + The default behaviour of automatically choosing a suitable default compressor remains if the compressor argument is not given. + To reproduce the behaviour in previous zarr-python versions when `compressor=None` was passed, pass `compressor='auto'` instead. ([#3039](https://github.com/zarr-developers/zarr-python/issues/3039)) + - Fixed the typing of `dimension_names` arguments throughout so that it now accepts iterables that contain `None` alongside `str`. ([#3045](https://github.com/zarr-developers/zarr-python/issues/3045)) - Using various functions to open data with `mode='a'` no longer deletes existing data in the store. ([#3062](https://github.com/zarr-developers/zarr-python/issues/3062)) - Internally use `typesize` constructor parameter for `numcodecs.blosc.Blosc` to improve compression ratios back to the v2-package levels. ([#2962](https://github.com/zarr-developers/zarr-python/issues/2962)) diff --git a/docs/user-guide/data_types.md b/docs/user-guide/data_types.md index ac966f885f..91f828a738 100644 --- a/docs/user-guide/data_types.md +++ b/docs/user-guide/data_types.md @@ -139,17 +139,17 @@ Zarr V3 introduced several key changes to how data types are represented: The basic data types are identified by strings like `"int8"`, `"int16"`, etc., and data types that require a configuration can be identified by a JSON object. - For example, this JSON object declares a datetime data type: - - ```json - { - "name": "numpy.datetime64", - "configuration": { - "unit": "s", - "scale_factor": 10 + For example, this JSON object declares a datetime data type: + + ```json + { + "name": "numpy.datetime64", + "configuration": { + "unit": "s", + "scale_factor": 10 + } } - } - ``` + ``` - Zarr V3 data types do not have endianness. This is a departure from Zarr V2, where multi-byte data types are defined with endianness information. Instead, Zarr V3 requires that the endianness diff --git a/docs/user-guide/v3_migration.md b/docs/user-guide/v3_migration.md index 6d9d516ebe..4d97963be2 100644 --- a/docs/user-guide/v3_migration.md +++ b/docs/user-guide/v3_migration.md @@ -42,40 +42,40 @@ the following actions in order: will be compatible in Zarr-Python 3. However, the following breaking API changes are planned: - - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs - directly from `numcodecs`: - - ```python exec="false" reason="intentionally shows the old/incorrect import for contrast" - from numcodecs import Blosc - # instead of: - # from zarr import Blosc - ``` - - - The `zarr.v3_api_available` feature flag is being removed. In Zarr-Python 3 - the v3 API is always available, so you shouldn't need to use this flag. - - `zarr.errors` has been consolidated. Several exception classes from - Zarr-Python 2 (such as `zarr.errors.PathNotFoundError`) have been removed - or replaced. For example, missing nodes now raise `zarr.errors.NodeNotFoundError` - (which subclasses both `BaseZarrError` and `FileNotFoundError`) instead of - `zarr.errors.PathNotFoundError`. Review any code that catches exceptions - from `zarr.errors` after migrating. - - The following internal modules are being removed or significantly changed. If - your application relies on imports from any of the below modules, you will need - to either a) modify your application to no longer rely on these imports or b) - vendor the parts of the specific modules that you need. - - * `zarr.attrs` has gone, with no replacement - * `zarr.codecs` has changed, see "Codecs" section below for more information - * `zarr.context` has gone, with no replacement - * `zarr.core` remains but should be considered private API - * `zarr.hierarchy` has gone, with no replacement (use `zarr.Group` in place of `zarr.hierarchy.Group`) - * `zarr.indexing` has gone, with no replacement - * `zarr.meta` has gone, with no replacement - * `zarr.meta_v1` has gone, with no replacement - * `zarr.sync` has gone, with no replacement - * `zarr.types` has gone, with no replacement - * `zarr.util` has gone, with no replacement - * `zarr.n5` has gone, see below for an alternative N5 option + - `numcodecs.*` will no longer be available in `zarr.*`. To migrate, import codecs + directly from `numcodecs`: + + ```python exec="false" reason="intentionally shows the old/incorrect import for contrast" + from numcodecs import Blosc + # instead of: + # from zarr import Blosc + ``` + + - The `zarr.v3_api_available` feature flag is being removed. In Zarr-Python 3 + the v3 API is always available, so you shouldn't need to use this flag. + - `zarr.errors` has been consolidated. Several exception classes from + Zarr-Python 2 (such as `zarr.errors.PathNotFoundError`) have been removed + or replaced. For example, missing nodes now raise `zarr.errors.NodeNotFoundError` + (which subclasses both `BaseZarrError` and `FileNotFoundError`) instead of + `zarr.errors.PathNotFoundError`. Review any code that catches exceptions + from `zarr.errors` after migrating. + - The following internal modules are being removed or significantly changed. If + your application relies on imports from any of the below modules, you will need + to either a) modify your application to no longer rely on these imports or b) + vendor the parts of the specific modules that you need. + + * `zarr.attrs` has gone, with no replacement + * `zarr.codecs` has changed, see "Codecs" section below for more information + * `zarr.context` has gone, with no replacement + * `zarr.core` remains but should be considered private API + * `zarr.hierarchy` has gone, with no replacement (use `zarr.Group` in place of `zarr.hierarchy.Group`) + * `zarr.indexing` has gone, with no replacement + * `zarr.meta` has gone, with no replacement + * `zarr.meta_v1` has gone, with no replacement + * `zarr.sync` has gone, with no replacement + * `zarr.types` has gone, with no replacement + * `zarr.util` has gone, with no replacement + * `zarr.n5` has gone, see below for an alternative N5 option 3. Test that your package works with version 3. 4. Update the pin to include `zarr>=3,<4`. @@ -123,8 +123,9 @@ The following sections provide details on breaking changes in Zarr-Python 3. 2. The h5py compatibility methods `create_dataset` and `require_dataset` have been removed. Use the following replacements: - - [`zarr.Group.create_array`][] in place of `Group.create_dataset` - - [`zarr.Group.require_array`][] in place of `Group.require_dataset` + - [`zarr.Group.create_array`][] in place of `Group.create_dataset` + - [`zarr.Group.require_array`][] in place of `Group.require_dataset` + 3. Disallow "." syntax for getting group members. To get a member of a group named `foo`, use `group["foo"]` in place of `group.foo`. 4. The `zarr.storage.init_group` low-level helper function has been removed. Use From 326545217ffe60112b2da1175a7423ff4acb7219 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 17 Jul 2026 13:18:56 +0200 Subject: [PATCH 07/16] chore(deps): bump the actions group across 1 directory with 6 updates (#4152) Bumps the actions group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `6.0.3` | `7.0.0` | | [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `8.2.0` | `8.3.2` | | [CodSpeedHQ/action](https://github.com/codspeedhq/action) | `4.18.1` | `4.18.5` | | [github-community-projects/issue-metrics](https://github.com/github-community-projects/issue-metrics) | `4.2.8` | `5.0.0` | | [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) | `2.8.0` | `2.9.0` | | [actions/labeler](https://github.com/actions/labeler) | `6.1.0` | `6.2.0` | Updates `actions/checkout` from 6.0.3 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v6.0.3...9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0) Updates `astral-sh/setup-uv` from 8.2.0 to 8.3.2 - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](https://github.com/astral-sh/setup-uv/compare/fac544c07dec837d0ccb6301d7b5580bf5edae39...11f9893b081a58869d3b5fccaea48c9e9e46f990) Updates `CodSpeedHQ/action` from 4.18.1 to 4.18.5 - [Release notes](https://github.com/codspeedhq/action/releases) - [Changelog](https://github.com/CodSpeedHQ/action/blob/main/CHANGELOG.md) - [Commits](https://github.com/codspeedhq/action/compare/a4a36bb07c0638b0b4ca52bf1f3dad1b4289e52f...f99becdce5e5d51fd556489ebef684f4ecfd6286) Updates `github-community-projects/issue-metrics` from 4.2.8 to 5.0.0 - [Release notes](https://github.com/github-community-projects/issue-metrics/releases) - [Commits](https://github.com/github-community-projects/issue-metrics/compare/44173f9e0a3b2144a777a10a340e4c09a25ac9f8...df8c49d20958f9345281fa2124858bd0ad227e1f) Updates `lycheeverse/lychee-action` from 2.8.0 to 2.9.0 - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](https://github.com/lycheeverse/lychee-action/compare/8646ba30535128ac92d33dfc9133794bfdd9b411...e7477775783ea5526144ba13e8db5eec57747ce8) Updates `actions/labeler` from 6.1.0 to 6.2.0 - [Release notes](https://github.com/actions/labeler/releases) - [Commits](https://github.com/actions/labeler/compare/f27b608878404679385c85cfa523b85ccb86e213...b8dd2d9be0f68b860e7dae5dae7d772984eacd6d) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/labeler dependency-version: 6.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: astral-sh/setup-uv dependency-version: 8.3.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: CodSpeedHQ/action dependency-version: 4.18.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: github-community-projects/issue-metrics dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: lycheeverse/lychee-action dependency-version: 2.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/check_changelogs.yml | 2 +- .github/workflows/codspeed.yml | 2 +- .github/workflows/docs.yml | 2 +- .github/workflows/downstream.yml | 4 ++-- .github/workflows/gpu_test.yml | 2 +- .github/workflows/hypothesis.yaml | 2 +- .github/workflows/issue-metrics.yml | 2 +- .github/workflows/links.yml | 4 ++-- .github/workflows/lint.yml | 2 +- .github/workflows/needs_release_notes.yml | 2 +- .github/workflows/test.yml | 8 ++++---- .github/workflows/zarr-metadata-release.yml | 2 +- .github/workflows/zarr-metadata.yml | 6 +++--- 13 files changed, 20 insertions(+), 20 deletions(-) diff --git a/.github/workflows/check_changelogs.yml b/.github/workflows/check_changelogs.yml index 25034b868d..0033b43db2 100644 --- a/.github/workflows/check_changelogs.yml +++ b/.github/workflows/check_changelogs.yml @@ -22,7 +22,7 @@ jobs: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Check zarr-python changelog entries run: uv run --no-sync python ci/check_changelog_entries.py diff --git a/.github/workflows/codspeed.yml b/.github/workflows/codspeed.yml index 1d8e847ec5..39cd8eb261 100644 --- a/.github/workflows/codspeed.yml +++ b/.github/workflows/codspeed.yml @@ -32,7 +32,7 @@ jobs: with: version: '1.16.5' - name: Run the benchmarks - uses: CodSpeedHQ/action@a4a36bb07c0638b0b4ca52bf1f3dad1b4289e52f # v4.18.1 + uses: CodSpeedHQ/action@f99becdce5e5d51fd556489ebef684f4ecfd6286 # v4.18.5 with: mode: walltime run: hatch run test.py3.12-minimal:pytest tests/benchmarks --codspeed diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml index c72b493b12..baf9233fc7 100644 --- a/.github/workflows/docs.yml +++ b/.github/workflows/docs.yml @@ -22,7 +22,7 @@ jobs: - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: persist-credentials: false - - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + - uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - run: uv sync --group docs # Fast source-level guards that need no built site, so they run before the (slower) # build for a quick failure: every public export is in the API reference, and no diff --git a/.github/workflows/downstream.yml b/.github/workflows/downstream.yml index d8aa2907be..f65f8d47e3 100644 --- a/.github/workflows/downstream.yml +++ b/.github/workflows/downstream.yml @@ -45,7 +45,7 @@ jobs: python-version: '3.13' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install xarray and test dependencies working-directory: xarray @@ -102,7 +102,7 @@ jobs: python-version: '3.13' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install numcodecs with test-zarr-main group working-directory: numcodecs diff --git a/.github/workflows/gpu_test.yml b/.github/workflows/gpu_test.yml index f2bf4907fa..bbbb3e5133 100644 --- a/.github/workflows/gpu_test.yml +++ b/.github/workflows/gpu_test.yml @@ -62,7 +62,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc with: diff --git a/.github/workflows/hypothesis.yaml b/.github/workflows/hypothesis.yaml index 01212dfb56..e836f30a5b 100644 --- a/.github/workflows/hypothesis.yaml +++ b/.github/workflows/hypothesis.yaml @@ -57,7 +57,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch uses: pypa/hatch@257e27e51a6a5616ed08a39a408a21c35c9931bc with: diff --git a/.github/workflows/issue-metrics.yml b/.github/workflows/issue-metrics.yml index 53860d21c8..adbd2748a0 100644 --- a/.github/workflows/issue-metrics.yml +++ b/.github/workflows/issue-metrics.yml @@ -33,7 +33,7 @@ jobs: echo "last_month=$first_day..$last_day" >> "$GITHUB_ENV" - name: Run issue-metrics tool - uses: github-community-projects/issue-metrics@44173f9e0a3b2144a777a10a340e4c09a25ac9f8 # v4.2.8 + uses: github-community-projects/issue-metrics@df8c49d20958f9345281fa2124858bd0ad227e1f # v5.0.0 env: GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} SEARCH_QUERY: 'repo:zarr-developers/zarr-python is:issue created:${{ env.last_month }} -reason:"not planned"' diff --git a/.github/workflows/links.yml b/.github/workflows/links.yml index a53de8da83..f606f12a3e 100644 --- a/.github/workflows/links.yml +++ b/.github/workflows/links.yml @@ -13,13 +13,13 @@ jobs: permissions: issues: write # required for peter-evans/create-issue-from-file steps: - - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 + - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 with: persist-credentials: false - name: Link Checker id: lychee - uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2.8.0 + uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2.9.0 with: fail: false diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 8190b1e061..dacba6648f 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -27,7 +27,7 @@ jobs: with: python-version: "3.12" - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: true - uses: j178/prek-action@e98a699c41eb69ab013a45817a0406469a748f8d # v2.0.5 diff --git a/.github/workflows/needs_release_notes.yml b/.github/workflows/needs_release_notes.yml index 1f79725b86..fa555d1478 100644 --- a/.github/workflows/needs_release_notes.yml +++ b/.github/workflows/needs_release_notes.yml @@ -21,7 +21,7 @@ jobs: pull-requests: write # Required to add labels to PRs runs-on: ubuntu-latest steps: - - uses: actions/labeler@f27b608878404679385c85cfa523b85ccb86e213 # v6.1.0 + - uses: actions/labeler@b8dd2d9be0f68b860e7dae5dae7d772984eacd6d # v6.2.0 with: repo-token: ${{ secrets.GITHUB_TOKEN }} sync-labels: true diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 7bc43512b5..ab78cfbe2b 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -66,7 +66,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Set Up Hatch Env @@ -115,7 +115,7 @@ jobs: python-version: ${{ matrix.python-version }} cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Set Up Hatch Env @@ -150,7 +150,7 @@ jobs: python-version: '3.13' cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Set Up Hatch Env @@ -174,7 +174,7 @@ jobs: python-version: '3.13' cache: 'pip' - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Install Hatch run: python -m pip install hatch==1.16.5 - name: Run Benchmarks diff --git a/.github/workflows/zarr-metadata-release.yml b/.github/workflows/zarr-metadata-release.yml index db05489798..5021f79d2e 100644 --- a/.github/workflows/zarr-metadata-release.yml +++ b/.github/workflows/zarr-metadata-release.yml @@ -51,7 +51,7 @@ jobs: path: dist - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: false diff --git a/.github/workflows/zarr-metadata.yml b/.github/workflows/zarr-metadata.yml index 95e8251227..4e5bb0fb1a 100644 --- a/.github/workflows/zarr-metadata.yml +++ b/.github/workflows/zarr-metadata.yml @@ -36,7 +36,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: true - name: Set up Python ${{ matrix.python-version }} @@ -58,7 +58,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 - name: Run ruff run: uvx ruff check . @@ -74,7 +74,7 @@ jobs: with: persist-credentials: false - name: Install uv - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # v8.2.0 + uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2 with: enable-cache: true - name: Set up Python From 50b7e016590d76011382771fd52843949782c9c0 Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Fri, 17 Jul 2026 15:34:21 +0200 Subject: [PATCH 08/16] Fix/memorystore buffer aliasing (#4157) * fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * fix(store): copy buffers on write in MemoryStore Encoding an uncompressed chunk hands the store a zero-copy view of the caller's array, and MemoryStore keeps whatever it is given alive in a dict rather than serializing it. Mutating the source array after a write therefore rewrote chunks already committed to the store, silently. Stores that serialize on write (LocalStore, ZipStore, remote stores) are unaffected, so they keep the full benefit of #3885. Only MemoryStore pays the copy, and only where it was aliasing to begin with: an uncompressed 34 MB write goes from ~23 ms to ~39 ms, while compressed writes are unchanged. Copying at the store boundary rather than narrowing the fast path in _merge_chunk_array also fixes the single-chunk case, which aliased in v3.2.1 too. Assisted-by: ClaudeCode:claude-opus-4.8 * docs: add changelog entry for MemoryStore buffer copy Assisted-by: ClaudeCode:claude-opus-4.8 * docs: correct changelog --- changes/4157.bugfix.md | 11 +++++++ src/zarr/core/buffer/core.py | 2 ++ src/zarr/storage/_memory.py | 24 ++++++++++++++-- tests/test_store/test_memory.py | 51 ++++++++++++++++++++++++++++++++- 4 files changed, 84 insertions(+), 4 deletions(-) create mode 100644 changes/4157.bugfix.md diff --git a/changes/4157.bugfix.md b/changes/4157.bugfix.md new file mode 100644 index 0000000000..6b0d0fcc67 --- /dev/null +++ b/changes/4157.bugfix.md @@ -0,0 +1,11 @@ +`MemoryStore` now copies buffers as they are written, so it never retains the +caller's memory. Previously an uncompressed write handed the store a zero-copy +view of the user's array, and mutating that array afterwards would silently +rewrite chunks already committed to the store. + +Only `MemoryStore` is affected: stores that serialize on write, such as +`LocalStore` and `ZipStore`, never aliased the caller's memory. Uncompressed +writes to a `MemoryStore` are correspondingly slower, since the copy that makes +the stored data independent is now actually performed; compressed writes are +unchanged. Buffers supplied through the `store_dict` argument remain the +caller's responsibility and are stored as-is. diff --git a/src/zarr/core/buffer/core.py b/src/zarr/core/buffer/core.py index 497543f88f..b8f7f11cd4 100644 --- a/src/zarr/core/buffer/core.py +++ b/src/zarr/core/buffer/core.py @@ -45,6 +45,8 @@ def __getitem__(self, key: slice) -> Self: ... def __setitem__(self, key: slice, value: Any) -> None: ... + def copy(self) -> Self: ... + @runtime_checkable class NDArrayLike(Protocol): diff --git a/src/zarr/storage/_memory.py b/src/zarr/storage/_memory.py index 5f5b632d4a..97dd355515 100644 --- a/src/zarr/storage/_memory.py +++ b/src/zarr/storage/_memory.py @@ -26,6 +26,18 @@ logger = getLogger(__name__) +def _copy_buffer(value: Buffer) -> Buffer: + """Copy `value` so the store does not retain the caller's memory. + + Encoding a chunk can hand the store a zero-copy view of the user's array + (an uncompressed write is the common case), and unlike stores that + serialize on write, this one keeps whatever it is given alive in a dict. + Without this copy a later mutation of the user's array would rewrite + chunks already committed to the store. + """ + return type(value).from_array_like(value.as_array_like().copy()) + + class MemoryStore(Store): """ Store for local memory. @@ -42,6 +54,12 @@ class MemoryStore(Store): supports_writes supports_deletes supports_listing + + Notes + ----- + Writes copy the buffer they are given, so the store never aliases the + caller's memory. Buffers passed via `store_dict` are the caller's + responsibility and are stored as-is. """ supports_writes: bool = True @@ -117,7 +135,7 @@ def set_sync(self, key: str, value: Buffer) -> None: raise TypeError( f"MemoryStore.set(): `value` must be a Buffer instance. Got an instance of {type(value)} instead." ) - self._store_dict[key] = value + self._store_dict[key] = _copy_buffer(value) def delete_sync(self, key: str) -> None: self._check_writable() @@ -178,13 +196,13 @@ async def set(self, key: str, value: Buffer, byte_range: tuple[int, int] | None buf[byte_range[0] : byte_range[1]] = value self._store_dict[key] = buf else: - self._store_dict[key] = value + self._store_dict[key] = _copy_buffer(value) async def set_if_not_exists(self, key: str, value: Buffer) -> None: # docstring inherited self._check_writable() await self._ensure_open() - self._store_dict.setdefault(key, value) + self._store_dict.setdefault(key, _copy_buffer(value)) async def delete(self, key: str) -> None: # docstring inherited diff --git a/tests/test_store/test_memory.py b/tests/test_store/test_memory.py index 35504718a7..36265423e6 100644 --- a/tests/test_store/test_memory.py +++ b/tests/test_store/test_memory.py @@ -8,7 +8,7 @@ import pytest import zarr -from zarr.core.buffer import Buffer, cpu, gpu +from zarr.core.buffer import Buffer, cpu, default_buffer_prototype, gpu from zarr.errors import ZarrUserWarning from zarr.storage import GpuMemoryStore, ManagedMemoryStore, MemoryStore from zarr.testing.store import StoreTests @@ -76,6 +76,55 @@ async def test_deterministic_size( np.testing.assert_array_equal(a[:3], 1) np.testing.assert_array_equal(a[3:], 0) + @pytest.mark.parametrize("method", ["set", "set_sync", "set_if_not_exists"]) + async def test_set_does_not_retain_caller_buffer(self, store: MemoryStore, method: str) -> None: + """Writing a buffer must not alias the caller's memory. + + MemoryStore keeps whatever it is handed alive in a dict, so retaining + the caller's buffer lets a later mutation of that buffer rewrite data + already committed to the store. + """ + source = np.frombuffer(bytearray(b"\x01\x02\x03\x04"), dtype="B") + value = cpu.Buffer.from_array_like(source) + + if method == "set_sync": + store.set_sync("k", value) + else: + await getattr(store, method)("k", value) + + source[:] = 0xF # mutate the caller's memory after the write + stored = await store.get("k", prototype=default_buffer_prototype()) + assert stored is not None + assert stored.to_bytes() == b"\x01\x02\x03\x04" + + @pytest.mark.parametrize( + "pipeline", + [ + "zarr.core.codec_pipeline.BatchedCodecPipeline", + "zarr.core.codec_pipeline.FusedCodecPipeline", + ], + ) + @pytest.mark.parametrize(("shape", "chunks"), [((30,), (10,)), ((8,), (4,)), ((4,), (4,))]) + def test_write_does_not_alias_source_array( + self, pipeline: str, shape: tuple[int], chunks: tuple[int] + ) -> None: + """Mutating the source array after a write must not corrupt stored chunks. + + Without compression the encoded buffer is a zero-copy view of the + caller's array all the way down to the store, so this covers both the + single-chunk and multi-chunk write paths. + """ + with zarr.config.set({"codec_pipeline.path": pipeline}): + array = zarr.create_array( + store=MemoryStore(), shape=shape, chunks=chunks, dtype="i4", compressors=None + ) + source = np.arange(shape[0], dtype="i4") + expected = source.copy() + array[:] = source + source[:] = -1 + + np.testing.assert_array_equal(array[:], expected) + # --- byte-range-write tests: disabled --- # Byte-range-write support (set_range / set_range_sync / SupportsSetRange) # was removed from this PR pending a decision on the store interface. These From bd0f1f1dcecc08c44477567f6048448ae1d7d1e3 Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Fri, 17 Jul 2026 17:26:39 +0200 Subject: [PATCH 09/16] Update roadmap.md Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> --- docs/roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 7a0c770b78..9f59e40187 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -46,7 +46,7 @@ directions: - Make Zarr-Python APIs ergonomic and useful for developers. - Expand our scope to cover vital quality-of-life routines like data copying, rechunking, and the like. -- Support the growth of Python tools across all levels of the Zarr stack. +- Ease the growth of Python tools across all levels of the Zarr stack. - Accelerate the implementation of new codecs, chunk grids, chunk key encodings, etc. From 641d5c092dcddb180f2067623e9ab9b26bb32702 Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Fri, 17 Jul 2026 17:27:25 +0200 Subject: [PATCH 10/16] Update roadmap.md Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> --- docs/roadmap.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 9f59e40187..e18f49d6f5 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -67,8 +67,8 @@ validator only needs to read metadata documents; a visualization tool may only need read-only array access; other tools need everything. We think of this as a "Zarr stack", from most abstract to most concrete: -1. **Conventions** — domain-specific schemas built on top of Zarr (OME-NGFF, - GeoZarr, anndata-zarr). +1. **Conventions** — application and/or domain-specific schemas built on top of Zarr (OME-NGFF, + GeoZarr, anndata-zarr, multiscales). 2. **Groups** — Zarr hierarchies, traversal, group-level attributes. 3. **Arrays** — the user-facing array object, plus indexing and slicing. 4. **Chunk decoding** — the codec pipeline. From 649f60b23cb02bdcb87fd069a1c0bce3b10d34c3 Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Fri, 17 Jul 2026 17:27:36 +0200 Subject: [PATCH 11/16] Update roadmap.md Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> --- docs/roadmap.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index e18f49d6f5..cbe7ce79d6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -37,7 +37,7 @@ call **"v4"** — is that overdue investment. If the 3.0 goals could be sloganized as "migrate to Zarr V3, and improve cloud storage support", the slogan for the v4 goals is: -**"support a Zarr-based Python ecosystem for chunked arrays"**. Zarr-Python +**"a frictionless Zarr-based Python ecosystem for chunked arrays"**. Zarr-Python should be *foundational* for the growing number of Python packages that work with data in the Zarr format. Concretely, that means pushing in these directions: From ab769985d7aa0279e9f628ab4148660bbf4921a5 Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Sun, 19 Jul 2026 14:19:48 -0400 Subject: [PATCH 12/16] chore: refine link checker configuration (#4158) --- lychee.toml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lychee.toml b/lychee.toml index 38b2b8ab7a..54a5b49b8d 100644 --- a/lychee.toml +++ b/lychee.toml @@ -1,6 +1,9 @@ # Configuration for the lychee link checker (https://lychee.cli.rs/). # Auto-discovered as ./lychee.toml by the lychee GitHub Action. +# Treat redirect status codes as success rather than failures. +accept = ["200..=299"] + # Files lychee should not scan for links. exclude_path = [ # mkdocs-material theme overrides: hrefs are Jinja expressions like @@ -17,4 +20,6 @@ exclude = [ # documentation of a command rather than a reachable link. '^https?://0\.0\.0\.0', '^https?://(localhost|127\.0\.0\.1)(:\d+)?', + # SPEC 0 page times out but is valid. + '^https://scientific-python\.org/specs/spec-0000', ] From ecc2d77718dbc4343da052347c4377945246d4fa Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Mon, 20 Jul 2026 16:39:17 +0200 Subject: [PATCH 13/16] fix(store): don't close a shared filesystem in FsspecStore.close() (#4165) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * fix(store): FsspecStore.close() no longer closes the filesystem FsspecStore.close() closed the underlying filesystem's session, on the premise that a store built by from_url "owns" the filesystem it created. That premise does not hold: fsspec caches and shares filesystem instances across callers (its instance cache keys on storage options, not path), and users can hand one filesystem to many stores directly. Closing one store therefore killed the session that sibling stores were still using, and left the dead filesystem in fsspec's cache for later callers. Determining whether a filesystem is actually shared requires reaching into fsspec's private instance cache (_cache, _fs_token, cachable) and walking wrapper chains for caching/proxy filesystems — an implementation detail that leaks upward and that we would have to keep in sync with fsspec forever, getting it subtly wrong in between. The wrapper case alone (simplecache::/dir://) already slipped through a cache-membership check. The filesystem's lifecycle is simply not the store's to manage. This removes the ownership model added in the unreleased gh-4003: no _owns_fs, no _close_fs, no ownership transfer in with_read_only, and close() just marks the store not-open. The only thing given up is suppressing an "Unclosed client session" ResourceWarning, which was true anyway — the session belongs to a cached filesystem that outlives the store. Since gh-4003 never shipped (latest release is v3.2.1), its changelog fragment is removed rather than superseded. Assisted-by: ClaudeCode:claude-opus-4.8 * test: skip with_read_only fs test when AsyncFileSystemWrapper is absent test_with_read_only_shares_filesystem replaced an ownership test that carried a guard for fsspec < 2024.12.0, and the guard was dropped in the rewrite. The test still opens a file:// URL, which needs AsyncFileSystemWrapper, so it failed the min_deps job. Assisted-by: ClaudeCode:claude-opus-4.8 * docs: correct changelog claim about gh-4003 release status The fragment said gh-4003 was unreleased with no net change for released versions. Its text is already in the staged 3.3.0 release notes, so the revert is a real behavior change for anyone relying on close() releasing the session. Assisted-by: ClaudeCode:claude-opus-4.8 * docs: remove changelog entry for unreleased versions --- src/zarr/storage/_fsspec.py | 70 ++++------------- tests/test_store/test_fsspec.py | 134 ++++++++------------------------ 2 files changed, 47 insertions(+), 157 deletions(-) diff --git a/src/zarr/storage/_fsspec.py b/src/zarr/storage/_fsspec.py index 617980ac19..37d134dd95 100644 --- a/src/zarr/storage/_fsspec.py +++ b/src/zarr/storage/_fsspec.py @@ -3,7 +3,6 @@ import json import warnings from contextlib import suppress -from logging import getLogger from typing import TYPE_CHECKING, Any from packaging.version import parse as parse_version @@ -19,8 +18,6 @@ from zarr.errors import ZarrUserWarning from zarr.storage._utils import _dereference_path -logger = getLogger(__name__) - if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterable @@ -38,26 +35,6 @@ ) -async def _close_fs(fs: AsyncFileSystem) -> None: - """ - Best-effort async close of an fsspec async filesystem owned by FsspecStore. - - For filesystems that expose `set_session()` (e.g. s3fs) the underlying - aiohttp `ClientSession` is closed explicitly, which prevents - "Unclosed client session" `ResourceWarning`s from aiohttp. For all - other filesystem types the call is a no-op (not every implementation - manages an HTTP session directly). - - Note that `set_session()` lazily creates a session if none exists yet, so - closing a store that never performed any I/O may instantiate a session - purely to close it. This is accepted best-effort behavior; fsspec does not - expose a stable, cross-implementation way to test for an existing session. - """ - if hasattr(fs, "set_session"): - session = await fs.set_session() - await session.close() - - def _make_async(fs: AbstractFileSystem) -> AsyncFileSystem: """Convert a sync FSSpec filesystem to an async FFSpec filesystem @@ -126,6 +103,15 @@ class FsspecStore(Store): ZarrUserWarning If the file system (fs) was not created with `asynchronous=True`. + Notes + ----- + Closing the store does not close the underlying filesystem or its network + session. fsspec caches and shares filesystem instances across callers, so + the store cannot know whether it is the only user, and closing a shared + session would break other stores. The filesystem's lifecycle belongs to + whoever created it; use fsspec's own tools (e.g. `clear_instance_cache`) + to release it. + See Also -------- FsspecStore.from_upath @@ -152,9 +138,6 @@ def __init__( self.fs = fs self.path = path self.allowed_exceptions = allowed_exceptions - # True only when this store created fs itself (from_url / from_mapper with new instance). - # Callers who supply their own fs remain responsible for its lifecycle. - self._owns_fs: bool = False if not self.fs.async_impl: raise TypeError("Filesystem needs to support async operations.") @@ -220,17 +203,13 @@ def from_mapper( ------- FsspecStore """ - original_fs = fs_map.fs - fs = _make_async(original_fs) - store = cls( + fs = _make_async(fs_map.fs) + return cls( fs=fs, path=fs_map.root, read_only=read_only, allowed_exceptions=allowed_exceptions, ) - # _make_async returns a new instance when converting sync→async; own it. - store._owns_fs = fs is not original_fs - return store @classmethod def from_url( @@ -272,39 +251,16 @@ def from_url( if not fs.async_impl: fs = _make_async(fs) - store = cls(fs=fs, path=path, read_only=read_only, allowed_exceptions=allowed_exceptions) - store._owns_fs = True - return store + return cls(fs=fs, path=path, read_only=read_only, allowed_exceptions=allowed_exceptions) def with_read_only(self, read_only: bool = False) -> FsspecStore: # docstring inherited - new_store = type(self)( + return type(self)( fs=self.fs, path=self.path, allowed_exceptions=self.allowed_exceptions, read_only=read_only, ) - # The derived store shares the same fs. Transfer ownership so the - # surviving store closes it, and clear ours to avoid a double-close. - # Otherwise the common `from_url(...).with_read_only()` chain would - # drop the only owner (the unreferenced source) and leak the session. - new_store._owns_fs = self._owns_fs - self._owns_fs = False - return new_store - - def close(self) -> None: - # docstring inherited - if self._owns_fs: - from zarr.core.sync import sync as zarr_sync - - # Best-effort: a failure to release the session must not block close(), - # but log it so a genuine regression in the close path stays observable - # rather than silently reverting to the leaking behavior. - try: - zarr_sync(_close_fs(self.fs)) - except Exception: - logger.debug("Failed to close owned filesystem %r", self.fs, exc_info=True) - super().close() async def clear(self) -> None: # docstring inherited diff --git a/tests/test_store/test_fsspec.py b/tests/test_store/test_fsspec.py index 898d49ec08..515e1526b6 100644 --- a/tests/test_store/test_fsspec.py +++ b/tests/test_store/test_fsspec.py @@ -276,75 +276,20 @@ async def test_delete_dir_unsupported_deletes(self, store: FsspecStore) -> None: ): await store.delete_dir("test_prefix") - # ── Filesystem lifecycle (ownership) ────────────────────────────────────── + # ── Filesystem lifecycle ────────────────────────────────────────────────── - def test_from_url_owns_filesystem(self, endpoint_url: str) -> None: - """FsspecStore.from_url() creates the async fs; it must own it.""" + async def test_close_marks_store_closed(self, endpoint_url: str) -> None: + """close() must succeed and mark the store not-open.""" store = FsspecStore.from_url( f"s3://{test_bucket_name}/lifecycle/", storage_options={"endpoint_url": endpoint_url, "anon": False}, ) - assert store._owns_fs - store.close() - - async def test_from_url_close_releases_store(self, endpoint_url: str) -> None: - """ - close() on a from_url() store must succeed without error and mark the - store as closed. For the owned filesystem, _close_fs() is invoked to - release the underlying S3 client / aiohttp connection pool. - """ - store = FsspecStore.from_url( - f"s3://{test_bucket_name}/lifecycle/", - storage_options={"endpoint_url": endpoint_url, "anon": False}, - ) - # Materialise the S3 client and connection pool. await store.set("probe", cpu.Buffer.from_bytes(b"x")) store.close() assert not store._is_open - def test_direct_construction_does_not_own_filesystem(self, endpoint_url: str) -> None: - """Direct FsspecStore() must not claim ownership — the caller owns the fs.""" - try: - from fsspec import url_to_fs - except ImportError: - from fsspec.core import url_to_fs - fs, path = url_to_fs( - f"s3://{test_bucket_name}", endpoint_url=endpoint_url, anon=False, asynchronous=True - ) - store = FsspecStore(fs=fs, path=path) - assert not store._owns_fs - - @pytest.mark.skipif( - parse_version(fsspec.__version__) < parse_version("2024.03.01"), - reason="Prior bug in from_upath", - ) - def test_from_upath_does_not_own_filesystem(self, endpoint_url: str) -> None: - """from_upath() uses the UPath's existing fs; the store must not own it.""" - upath = pytest.importorskip("upath") - path = upath.UPath( - f"s3://{test_bucket_name}/foo/bar/", - endpoint_url=endpoint_url, - anon=False, - asynchronous=True, - ) - store = FsspecStore.from_upath(path) - assert not store._owns_fs - - def test_from_mapper_does_not_own_already_async_filesystem(self, endpoint_url: str) -> None: - """from_mapper() with an already-async fs must not claim ownership.""" - s3_filesystem = s3fs.S3FileSystem( - asynchronous=True, - endpoint_url=endpoint_url, - anon=False, - skip_instance_cache=True, - ) - mapper = s3_filesystem.get_mapper(f"s3://{test_bucket_name}/") - store = FsspecStore.from_mapper(mapper) - # _make_async returns the same instance for an already-async fs. - assert not store._owns_fs - def array_roundtrip(store: FsspecStore) -> None: """ @@ -574,47 +519,47 @@ def test_open_s3map_raises(endpoint_url: str) -> None: zarr.open(store=mapper, storage_options={"anon": True}, mode="w", shape=(3, 3)) -async def test_close_fs_closes_s3_client() -> None: - """ - _close_fs() must call set_session() and then close() on the returned - S3 client. This is verified with mocks to avoid a real S3 connection. - """ - from unittest.mock import AsyncMock +async def test_close_does_not_close_filesystem_session() -> None: + """close() must not touch the filesystem's session. - from zarr.storage._fsspec import _close_fs + fsspec caches and shares filesystem instances across callers, so the + session is not the store's to close. HTTP is used because its aiohttp + session is observably closed for good; s3fs transparently reconnects, which + would hide a regression. No request is issued — set_session() only + constructs the session. + """ + pytest.importorskip("aiohttp") + store = FsspecStore.from_url("http://example.com/a") + session = await store.fs.set_session() - mock_client = AsyncMock() - mock_fs = AsyncMock() - mock_fs.set_session = AsyncMock(return_value=mock_client) + store.close() - await _close_fs(mock_fs) + assert not session.closed - mock_fs.set_session.assert_called_once() - mock_client.close.assert_called_once() +async def test_close_does_not_break_a_sibling_store() -> None: + """Closing one store must not close a session another store is using. -async def test_close_fs_no_op_for_fs_without_set_session() -> None: - """_close_fs() must be a no-op for filesystems that don't expose set_session().""" - from unittest.mock import AsyncMock + Two stores from different URLs on one host are handed the same cached + filesystem; a store that closed it on close() would take the sibling's + session down too. This is the regression guard for that bug. + """ + pytest.importorskip("aiohttp") + s1 = FsspecStore.from_url("http://example.com/a") + s2 = FsspecStore.from_url("http://example.com/b") + session = await s2.fs.set_session() - from zarr.storage._fsspec import _close_fs + s1.close() - mock_fs = AsyncMock(spec=[]) # empty spec — no set_session attribute - await _close_fs(mock_fs) # must not raise + assert not session.closed @pytest.mark.skipif( parse_version(fsspec.__version__) < parse_version("2024.12.0"), reason="No AsyncFileSystemWrapper", ) -def test_from_mapper_owns_wrapped_sync_filesystem(tmp_path: pathlib.Path) -> None: - """ - from_mapper() with a sync fs must wrap it in AsyncFileSystemWrapper and - claim ownership so that close() cleans it up. - - The local filesystem is synchronous; _make_async() produces a new - AsyncFileSystemWrapper instance — a different object from the original fs. - """ +def test_from_mapper_wraps_sync_filesystem(tmp_path: pathlib.Path) -> None: + """from_mapper() with a sync fs wraps it in an AsyncFileSystemWrapper.""" import fsspec as _fsspec from fsspec.implementations.asyn_wrapper import AsyncFileSystemWrapper @@ -622,32 +567,21 @@ def test_from_mapper_owns_wrapped_sync_filesystem(tmp_path: pathlib.Path) -> Non mapper = fs.get_mapper(str(tmp_path)) store = FsspecStore.from_mapper(mapper) assert isinstance(store.fs, AsyncFileSystemWrapper) - assert store._owns_fs @pytest.mark.skipif( parse_version(fsspec.__version__) < parse_version("2024.12.0"), reason="No AsyncFileSystemWrapper", ) -def test_with_read_only_transfers_filesystem_ownership(tmp_path: pathlib.Path) -> None: - """ - with_read_only() must transfer fs ownership to the derived store and clear - it on the source, so the surviving store closes the shared fs exactly once. - - In the common ``from_url(...).with_read_only()`` chain the source store is - immediately unreferenced; if ownership were not transferred, the only owner - would be garbage-collected without close() and the session would leak. - """ +def test_with_read_only_shares_filesystem(tmp_path: pathlib.Path) -> None: + """with_read_only() returns a store sharing the source's filesystem.""" source = FsspecStore.from_url(f"file://{tmp_path}", storage_options={"auto_mkdir": False}) - assert source._owns_fs derived = source.with_read_only(read_only=True) - # Ownership moved to the survivor; the source no longer owns it (no double-close). - assert derived._owns_fs - assert not source._owns_fs - # The derived store shares the same underlying fs. assert derived.fs is source.fs + assert derived.read_only + assert not source.read_only @pytest.mark.parametrize("asynchronous", [True, False]) From 8072b42fe9dd42074729f9fd35943f1be308b7fd Mon Sep 17 00:00:00 2001 From: Davis Bennett Date: Tue, 21 Jul 2026 20:31:56 +0200 Subject: [PATCH 14/16] Update docs/roadmap.md Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> --- docs/roadmap.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index cbe7ce79d6..10bea05768 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -154,11 +154,16 @@ before/after numbers. ### Lazy indexing -`Array.__getitem__` performs IO eagerly and returns NumPy arrays, which makes Zarr -arrays the odd one out among modern array libraries and blocks compliance with -the [Python Array API](https://data-apis.org/array-api/) standard. Add an -opt-in `array.lazy[...]` accessor backed by a stable coordinate-mapping algebra -(the `IndexTransform` work in +The Zarr-Python Array API was initially designed to mirror NumPy, with eager +syntax. `Array.__getitem__` performs IO eagerly and returns a NumPy arrays. +That was helpful to the dominant use-case at the time of its creation, but it +means deferred I/O and computation currently require an external library +such as Dask. It means there is no build-in support for representing multi +step reads as a single deferred plan. Further, it means that every chained +selection round-trips to storage independently. + +To solve this limitation, Add an opt-in `array.lazy[...]` accessor backed by a +stable coordinate-mapping algebra (the `IndexTransform` work in [#3906](https://github.com/zarr-developers/zarr-python/pull/3906)), plus a small query planner that turns chained selections into a single IO plan before any chunks are fetched. No new array type is introduced. Whether the *default* From e0623e1166c9d504af1d331c5b3a4ed2d9a88650 Mon Sep 17 00:00:00 2001 From: Davis Vann Bennett Date: Tue, 21 Jul 2026 20:42:42 +0200 Subject: [PATCH 15/16] docs: remove stale roadmap.md redirect so the new roadmap page renders The mkdocs-redirects plugin was generating a redirect stub for roadmap.md pointing at the old v3.0.8 docs, which clobbered the new roadmap page added in this PR. The developers/roadmap.html redirect is kept so old links to the historical v3 design roadmap still resolve. Assisted-by: ClaudeCode:claude-fable-5 --- mkdocs.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/mkdocs.yml b/mkdocs.yml index 25516a56dd..767c177118 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -202,7 +202,6 @@ plugins: 'search.html.md': 'index.md' 'tutorial.md': 'user-guide/installation.md' 'getting-started.md': 'quick-start.md' - 'roadmap.md': 'https://zarr.readthedocs.io/en/v3.0.8/developers/roadmap.html' 'installation.md': 'user-guide/installation.md' 'release.md': 'release-notes.md' 'about.html.md': 'index.md' From c0e1f1fd8d38c858459d055dae0a435af06ec61f Mon Sep 17 00:00:00 2001 From: Max Jones <14077947+maxrjones@users.noreply.github.com> Date: Tue, 21 Jul 2026 19:35:57 -0400 Subject: [PATCH 16/16] Rename +roadmap.doc.md to 4149.doc.md --- changes/{+roadmap.doc.md => 4149.doc.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename changes/{+roadmap.doc.md => 4149.doc.md} (100%) diff --git a/changes/+roadmap.doc.md b/changes/4149.doc.md similarity index 100% rename from changes/+roadmap.doc.md rename to changes/4149.doc.md