Track zarr v3 migration for OpenGHG storage

[brendan-m-murphy]

Umbrella tracker for the zarr v3 migration work.

Goal

Make OpenGHG's openghg.storage.ZarrStore work with zarr-python v3 while preserving current zarr v2 behavior during the transition.

Global context

The migration spans several cross-library areas that should be pinned down with focused tests before loosening OpenGHG's zarr dependency:

• zarr store APIs;
• xarray write semantics;
• chunk alignment;
• consolidated metadata;
• version copying.

Keep early work behavior-preserving where possible. Do not enable zarr v3 storage by default until the compatibility, copy, write-safety, metadata, and test-matrix pieces are in place.

References

• Zarr issue: [v3] Implement zarr.copy_store zarr-developers/zarr-python#2407
• Zarr Group.copy_to PR draft: Adding copy_store convenience method for Group zarr-developers/zarr-python#3612
• Zarr v3 migration guide: https://zarr.readthedocs.io/en/latest/user-guide/v3_migration/
• Xarray Dataset.to_zarr docs: https://docs.xarray.dev/en/v2025.06.1/generated/xarray.Dataset.to_zarr.html

Sub-issues

Use this checklist to track progress and, if needed, manually assign these as child issues of this umbrella issue.

• Add a local zarr compatibility layer #1659 - Add a local zarr compatibility layer
• Implement an OpenGHG copy_zarr_store helper #1660 - Implement an OpenGHG copy_zarr_store helper
• Fix versioned zarr store copy semantics #1661 - Fix versioned zarr store copy semantics
• Remove zarr ThreadSynchronizer use and adopt xarray chunk alignment #1662 - Remove zarr ThreadSynchronizer use and adopt xarray chunk alignment
• Make non-contiguous update writes structurally safe #1663 - Make non-contiguous update writes structurally safe
• Make consolidated metadata behavior zarr-format aware #1664 - Make consolidated metadata behavior zarr-format aware
• Add a zarr v3 test matrix for OpenGHG storage #1665 - Add a zarr v3 test matrix for OpenGHG storage

Suggested implementation order

1. Add a local zarr compatibility layer #1659
2. Implement an OpenGHG copy_zarr_store helper #1660
3. Fix versioned zarr store copy semantics #1661
4. Remove zarr ThreadSynchronizer use and adopt xarray chunk alignment #1662
5. Make non-contiguous update writes structurally safe #1663
6. Make consolidated metadata behavior zarr-format aware #1664
7. Add a zarr v3 test matrix for OpenGHG storage #1665

Agent update instructions

Agents working on any sub-issue should add a comment to this umbrella issue before handing off or closing their work. The comment should include:

• the sub-issue number worked on;
• what was changed;
• tests/checks run;
• any blockers, follow-up work, dependency constraints, or behavior that could affect later stages of the migration.

Keep this issue focused on progress tracking and cross-stage handoff notes.

[brendan-m-murphy]

Sub-issue worked on: #1659 - Add a local zarr compatibility layer

What changed:

• Added openghg/storage/_zarr_compat.py with zarr version/capability checks, memory/local store factories, store emptiness checks, key iteration, prefix-safe clearing, byte-size calculation, and a copy_store wrapper.
• Updated openghg/storage/_zarr_store.py to use the compatibility helpers instead of direct zarr.convenience, private zarr._storage.store.Store, and direct zarr.DirectoryStore / zarr.MemoryStore factory references.
• Added focused compat tests, including v2 store behavior and an async v3-style prefix-listing fake to guard against sibling-prefix overmatching.

Tests/checks run:

• .venv/bin/python -m pytest tests/storage/test_zarr_compat.py tests/storage/test_store.py tests/store/test_datasource.py -> 107 passed, 1 xfailed.
• .venv/bin/python -m black --check openghg/storage/_zarr_compat.py openghg/storage/_zarr_store.py tests/storage/test_zarr_compat.py -> passed.
• .venv/bin/python -m flake8 openghg/storage/_zarr_compat.py openghg/storage/_zarr_store.py tests/storage/test_zarr_compat.py -> passed.
• .venv/bin/python -m mypy --python-version 3.10 openghg/storage/_zarr_compat.py openghg/storage/_zarr_store.py -> passed.
• uv run ruff check ... and uv run ruff format --check ... could not run because ruff is not installed in the locked environment.

Follow-up notes for later migration stages:

• copy_store currently delegates to zarr.copy_store when available and raises NotImplementedError otherwise; Implement an OpenGHG copy_zarr_store helper #1660 should replace this with the OpenGHG copy helper for zarr v3.
• zarr.ThreadSynchronizer() remains in _zarr_store.py; Remove zarr ThreadSynchronizer use and adopt xarray chunk alignment #1662 is expected to remove that and adopt xarray chunk-alignment behavior.
• Zarr v3 storage is not enabled by default; this is behavior-preserving for the pinned zarr v2 environment.