Create read only copy if needed when opening a store path by maxrjones · Pull Request #3156 · zarr-developers/zarr-python

maxrjones · 2025-06-19T16:21:19Z

This PR makes a read-only copy of a store when using StorePath.open(mode="r").

See discussion in #3068. This augments #3068 (it still errors when attempting to open a writeable store path using a read-only store). This is an alternative to #3155.

Closes #3147

TODO:

Add unit tests and/or doctests in docstrings
Add docstrings and API docs for any new/modified user-facing classes and functions
New/modified features documented in docs/user-guide/*.rst
Changes documented as a new file in changes/
GitHub Actions have all passed
Test coverage is 100% (Codecov passes)

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

dstansby

Looks good to me. Requesting changes because:

This should get a release note
My old release note from when I added the error should be deleted

Co-authored-by: David Stansby <dstansby@gmail.com>

maxrjones · 2025-06-20T12:26:07Z

This should get a release note
My old release note from when I added the error should be deleted

Thanks, done in 10e50e4 (#3156)

This reverts commit 7ad760f.

TomNicholas · 2025-06-24T19:39:54Z

FYI I tried running xarray's zarr tests with this PR and while it fixes the basic read_only failure that #3068 caused (see for example this test, which passes with this PR), I still see a lot of other errors. I'm just trying to work out if those errors are caused by this PR or by zarr's dtypes refactor.

I also see a lot of things like this

FAILED xarray/tests/test_backends.py::TestZarrDictStore::test_warning_on_bad_chunks[2] - AssertionError: Got 1 unexpected warning(s): ["UserWarning('Store is not read-only but mode is 'r'. Creating a read-only copy. This behavior may change in the future with a more granular permissions model.'))"]

suggesting that xarray might need to swallow the warning being raised in this PR.

d-v-b · 2025-06-24T19:43:55Z

Why do we need to raise a warning here? Doesn't the new with_read_only method fully legitimize the creation of a new store in read-only mode?

TomNicholas · 2025-06-24T19:46:00Z

I agree, this isn't something to warn for, it's just a change of behaviour to be documented.

rabernat · 2025-06-24T20:00:33Z

I agree, this isn't something to warn for, it's just a change of behaviour to be documented.

👍 to this. FWIW I think we have been way to eager to issue warnings in Zarr. No warning is needed here.

TomNicholas · 2025-06-24T20:01:11Z

I'm just trying to work out if those errors are caused by this PR or by zarr's dtypes refactor.

After talking to @ianhi I think those other errors are related to the dtypes changes rather than to the changes in this PR.

…opers#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c)

) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py * Update src/zarr/storage/_common.py * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening * Improve code coverage --------- (cherry picked from commit 5731c6c) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io>

* Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * release notes --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io>

* Enable tests on 3.0.x branch (#3135) * Changelog for 3.0.9 * Backport PR #3149: Add GroupNotFound error to API docs (#3179) Co-authored-by: David Stansby <dstansby@gmail.com> * Backport PR #3140: Suppress FileNotFoundError when deleting keys in the obstore adapter (#3180) Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> * Backport PR #3138: Add with_read_only() convenience method to store (#3181) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * Create read only copy if needed when opening a store path (#3156) (#3182) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py * Update src/zarr/storage/_common.py * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening * Improve code coverage --------- (cherry picked from commit 5731c6c) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) (#3203) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- (cherry picked from commit 5a24487) Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) * Auto backport of pr 3195 on 3.0.10 (#3204) * Remove breaking check about `auto_mkdir` for FSSpecStore (#3193) * Remove breaking check from _make_async * Update expected error * Change import structure to protect against AttributeError * changelog * add test to ensure that we can create a read-only copy of the store with auto_mkdir=False * only test if the async wrapper is available --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> (cherry picked from commit 5a24487) * Add missing import for AsyncFileSystemWrapper for `_make_async` in `_fsspec.py` (#3195) * Add missing import for AsyncFileSystemWrapper in `_fsspec.py` * Add missing changelog entry for AsyncFileSystemWrapper import fix * Move AsyncFileSystemWrapper import past the version check in `_fsspec.py` * Add newline after AsyncFileSystemWrapper import in `_fsspec.py` * Simplify import statement for AsyncFileSystemWrapper in `_fsspec.py` --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com> (cherry picked from commit 97aa42f) --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com> * 3.0.9 release notes (#3183) * Create read only copy if needed when opening a store path (#3156) * Create read only copy if needed when opening a store path * Add ValueError to Raises section * Update expected warning * Update src/zarr/storage/_common.py Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> * Use ANY_ACCESS_MODE * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update src/zarr/storage/_common.py Co-authored-by: David Stansby <dstansby@gmail.com> * Update changes * Try using get_args on definition * Revert "Try using get_args on definition" This reverts commit 7ad760f. * Add test * Remove warning * Apply suggestion for try; except shortening Co-authored-by: Tom Nicholas <tom@earthmover.io> * Improve code coverage --------- Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> (cherry picked from commit 5731c6c) * release notes --------- Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: David Stansby <dstansby@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> * release notes --------- Co-authored-by: Lumberbot (aka Jack) <39504233+meeseeksmachine@users.noreply.github.com> Co-authored-by: Lukas Bindreiter <lukas.bindreiter@tilebox.com> Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com> Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com> Co-authored-by: Tom Nicholas <tom@earthmover.io> Co-authored-by: Altay Sansal <tasansal@users.noreply.github.com> Co-authored-by: Altay Sansal <altay.sansal@tgs.com>

codecov · 2025-09-23T00:00:20Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.47%. Comparing base (1d0ea2b) to head (07bb607).
⚠️ Report is 130 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3156   +/-   ##
=======================================
  Coverage   94.47%   94.47%           
=======================================
  Files          78       78           
  Lines        8578     8585    +7     
=======================================
+ Hits         8104     8111    +7     
  Misses        474      474

Files with missing lines	Coverage Δ
src/zarr/core/common.py	`90.47% <100.00%> (+0.11%)`	⬆️
src/zarr/storage/_common.py	`92.70% <100.00%> (+0.23%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Create read only copy if needed when opening a store path

6f52bfb

github-actions Bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jun 19, 2025

Add ValueError to Raises section

737ff9b

maxrjones commented Jun 19, 2025

View reviewed changes

Comment thread src/zarr/storage/_common.py

maxrjones commented Jun 19, 2025

View reviewed changes

Comment thread src/zarr/storage/_common.py

maxrjones commented Jun 19, 2025

View reviewed changes

Comment thread src/zarr/storage/_common.py

Update expected warning

957cd64

d-v-b reviewed Jun 19, 2025

View reviewed changes

Comment thread src/zarr/storage/_common.py Outdated

d-v-b reviewed Jun 19, 2025

View reviewed changes

Comment thread src/zarr/storage/_common.py Outdated

maxrjones and others added 2 commits June 19, 2025 13:19

Update src/zarr/storage/_common.py

23f4376

Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>

Use ANY_ACCESS_MODE

a42dc7c

maxrjones mentioned this pull request Jun 19, 2025

Only warn if opening writeable store with mode='r' #3155

Closed

dstansby previously requested changes Jun 20, 2025

View reviewed changes

Comment thread src/zarr/core/common.py

Comment thread src/zarr/storage/_common.py

Comment thread src/zarr/storage/_common.py Outdated

Comment thread src/zarr/storage/_common.py Outdated

maxrjones and others added 3 commits June 20, 2025 08:21

Update src/zarr/storage/_common.py

aba4919

Co-authored-by: David Stansby <dstansby@gmail.com>

Update src/zarr/storage/_common.py

0129458

Co-authored-by: David Stansby <dstansby@gmail.com>

Update changes

10e50e4

github-actions Bot removed the needs release notes Automatically applied to PRs which haven't added release notes label Jun 20, 2025

maxrjones added 3 commits June 20, 2025 08:26

Merge branch 'main' into maybe-copy-store

15becdf

Try using get_args on definition

7ad760f

Revert "Try using get_args on definition"

a1949f1

This reverts commit 7ad760f.

maxrjones requested a review from dstansby June 20, 2025 13:41

Add test

202d606

TomNicholas reviewed Jun 24, 2025

View reviewed changes

Comment thread src/zarr/storage/_common.py Outdated

Comment thread src/zarr/storage/_common.py Outdated

Comment thread src/zarr/storage/_common.py

Merge branch 'main' into maybe-copy-store

4d8e7ad

d-v-b removed the Still Needs Manual Backport label Jun 30, 2025

d-v-b mentioned this pull request Jun 30, 2025

3.0.9 release notes #3183

Merged

This was referenced Jun 30, 2025

⚠️ Nightly upstream-dev CI failed ⚠️ pydata/xarray#10261

Closed

ODD PI 25.3 Objective 5: Foundational Zarr-Python and Xarray Contributions NASA-IMPACT/veda-odd#165

Closed

maxrjones mentioned this pull request Jul 6, 2025

Use a release checklist and narrative announcement for breaking changes #3209

Closed

This was referenced Jul 23, 2026

Link Checker Report #4184

Open

Link Checker Report d-v-b/zarr-python#239

Open

Link Checker Report #4185

Open

Link Checker Report d-v-b/zarr-python#240

Open

Link Checker Report #4186

Open

Link Checker Report d-v-b/zarr-python#243

Open

Uh oh!

Uh oh!

Conversation

maxrjones commented Jun 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dstansby left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maxrjones commented Jun 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomNicholas commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

d-v-b commented Jun 24, 2025

Uh oh!

TomNicholas commented Jun 24, 2025

Uh oh!

rabernat commented Jun 24, 2025

Uh oh!

TomNicholas commented Jun 24, 2025

Uh oh!

codecov Bot commented Sep 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

maxrjones commented Jun 19, 2025 •

edited

Loading

TomNicholas commented Jun 24, 2025 •

edited

Loading

codecov Bot commented Sep 23, 2025 •

edited

Loading