🌳 ODD PI 26.4 Objective 4: Performance improvements for the Python Zarr ecosystem

[abarciauskas-bgse]

Motivation

NASA relies on zarr-python for virtual data access; performance improvements will improve developer experience and reduce compute costs. We have been building a strong foundation for zarr ecosystem through making Zarr-Python bindings to zarrs (Davis has explored). Zarrs has become a mature Zarr implementation. In other previous Rust-Python bindings, like Async-TIFF and Obstore, we have found large performance improvements. We’ve also learned how to build clean, maintainable Python bindings for Rust projects.

Description

In this objective, we will continue to work on performance improvements to Zarr-Python through integration its integration of rust libraries zarrs and zarrista.

Acceptance Criteria

• PR for zarrista into zarr-python
• performance changes documented

Sub-tasks

• Performance improvements for the Python Zarr ecosystem, building on Rust Zarrs library, with Zarrista
• Explore opt-in integration into Zarr-Python
• Work on benchmarking Zarrista
• Support downstream adoption (titiler and xarray) of lazy indexing
• Explore configuration profiles for easy, fast cloud operations
• Document performance improvements

LOE

1 FTE (split between Kyle and Davis)

Dependencies

None