ODD PI 25.3 Objective 5: Foundational Zarr-Python and Xarray Contributions

[abarciauskas-bgse]

Motivation

Zarr-Python and Xarray are foundational to many visualization and analysis services we wish to provide on top of NASA data. Zarr-Python 3 includes significant advancements in the Zarr specification and performance, but lacks some of the functionality of Zarr-Python 2.

Description

Provide critical contributions to zarr-python and xarray.

Acceptance Criteria

• Support implicit coordinates/flexible indices by contributing upstream requirements in Xarray and GeoZarr (note: this work also supports swath data) (Support flexible index usage with Zarr and Xarray #125)
• Support datetime64 and other data types in zarr-python 3
• Improved codecs handling in zarr-python ecosystem

Tasks

• Simplify refactor v3 data types zarr-developers/zarr-python#2874 to only include the new abstraction of existing dtypes.
• Implement datetime64 support for Zarr Python via a specification PR against the zarr-extensions repository (e.g., feature(data-types): add data-types from https://github.com/zarr-developers/zarr-python/pull/2874 zarr-developers/zarr-extensions#5), and an implementation PR in zarr-python.
• Coordinate with EODC team and NASA GES DISC about release timeline for the support of datetime64 arrays.
• Propose additional data types in Zarr Python from existing implementation, aiming for feature parity with Zarr V2
• Implement upcasting of Zarr V2 codecs to zarr v3 codecs
• Provide guidance on imagecodecs extension points in Zarr Python 3 / Zarr format 3
• Coordinate with Zarr maintainers and community on testing and release of codec and data type support

Stretch:

Produce a design document and implementation plan for variable length chunking support in Zarr Python 3

[abarciauskas-bgse]

Max is supporting the presentation on implicit coordinates in xarray (geozarr example)

[maxrjones]

Commits by me between April 1 and June 30 or Davis between April 25 and June 30 (acquired through GitHub API search e.g., https://github.com/zarr-developers/zarr-python/commits/main/?since=2025-04-25&until=2025-06-30&author=maxrjones):

Zarr-Python

• Create read only copy if needed when opening a store path zarr-developers/zarr-python#3156
• Disable codecov PR annotations zarr-developers/zarr-python#3166
• Add with_read_only() convenience method to store zarr-developers/zarr-python#3138
• Support async FSMap objects in zarr.open zarr-developers/zarr-python#2774
• Use context manager for temporary directories in store tests zarr-developers/zarr-python#3110
• Add more unit tests for GPU buffer zarr-developers/zarr-python#2978
• Fix hatch matrix setup for minimal and optional dependencies zarr-developers/zarr-python#2872
• simplify NDBuffer.as_scalar zarr-developers/zarr-python#3027
• fix/unbreak chunks initialized zarr-developers/zarr-python#2862
• add fill_value output to info zarr-developers/zarr-python#3081
• remove insertion of vlen-string codec for v2 metadata creation zarr-developers/zarr-python#3100
• remove fsspec from upstream tests zarr-developers/zarr-python#3116
• refactor v3 data types zarr-developers/zarr-python#2874
• string dtype fixes zarr-developers/zarr-python#3170
• Fix/windows int32 issue zarr-developers/zarr-python#3151

Zarr-extensions

• Fix links and add link-checking CI workflow zarr-developers/zarr-extensions#19

Rasterix

• https://github.com/dcherian/rasterix/pull/23

These counts don't include reviews of PRs and issues which often take more time and more strongly limit the progress of a library since fewer people have the requisite knowledge and permissions.

[abarciauskas-bgse]

closing as completed