From 0dd8b1ad449743a3ef279d02fcd1c29a6d5f5a15 Mon Sep 17 00:00:00 2001 From: ruaridhg Date: Wed, 17 Sep 2025 15:52:39 +0100 Subject: [PATCH 1/5] Add summary for benchmarking for Zarr --- README.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/README.md b/README.md index f3f0b27..4b0d755 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,30 @@ This textbook gives scientists: - a guide to designing parallel processing algorithms to work efficiently with chunked datasets - a guide to exporting chunked datasets to other 'tradditional' datasets +## Benchmarking for Zarr + +We created a set of benchmarks for writing data to Zarr with a range of different configurations as guidance for the options available when reading and writing 3D imaging data. +The different parameters were: + +- Type of image + - Heart: HiP-CT scan of a heart from the Human Organ Atlas + - Dense: segmented neurons from electron microscopy + - Sparse: A few select segmented neurons from electron microscopy +- Software libraries + - Tensorstore (fastest for both reading and writing data) + - zarr-python version 3 + - zarr-python version 2 (slowest for both reading and writing data) +- Compressor + - blosc-zstd provides the best compression ratio, for image and segmentation data. (options were blosc-blosclz, blosc-lz4, blosc-lz4hc, blosc-zlib, blosc-zstd as well as gzip and zstd) +- Compression level + - Setting compression levels beyond ~3 results in slightly better data compression but much longer write times. Compression level does not affect read time. +- Shuffle + - Setting the shuffle option increases data compression with no adverse effect on read/write times (shuffle, bitshuffle and noshuffle were the 3 options) +- Zarr format version + - There was no noticeable difference between Zarr format 2 and Zarr format 3 data +- Chunk size + - Setting a low chunk size (below around 90) has an adverse effect on read and write times. + ## Tools for working with chunked datasets ## Improvements to cloud visualisation From 9edbb8b5e5803ad095deda05da04fe5882ca53fe Mon Sep 17 00:00:00 2001 From: ruaridhg Date: Thu, 18 Sep 2025 11:45:52 +0100 Subject: [PATCH 2/5] Add summary for tools for working with chunked datasets --- README.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/README.md b/README.md index 4b0d755..56b73f3 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,11 @@ The different parameters were: ## Tools for working with chunked datasets +Contributions have been made to the zarr-python reposistory: + +- [Add CLI for converting v2 metadata to v3](https://github.com/zarr-developers/zarr-python/pull/3257) +- [Added ArrayNotFoundError](https://github.com/zarr-developers/zarr-python/pull/3367) + ## Improvements to cloud visualisation ## Acknowledgements From c7393b6a47103acd3ca702144fc973f25c37e9cc Mon Sep 17 00:00:00 2001 From: ruaridhg Date: Thu, 18 Sep 2025 13:35:50 +0100 Subject: [PATCH 3/5] Add summary for tools for working with chunked datasets --- README.md | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 56b73f3..6192977 100644 --- a/README.md +++ b/README.md @@ -39,11 +39,22 @@ The different parameters were: ## Tools for working with chunked datasets -Contributions have been made to the zarr-python reposistory: +Contributions have been made to the zarr-python repository: - [Add CLI for converting v2 metadata to v3](https://github.com/zarr-developers/zarr-python/pull/3257) - [Added ArrayNotFoundError](https://github.com/zarr-developers/zarr-python/pull/3367) +PRs have been opened in the zarr-python repository: + +- [Prevent creation of arrays/groups under a parent array](https://github.com/zarr-developers/zarr-python/pull/3407) +- [Holding space - Better document acceptable values for StoreLike] +- [Holding space - LRUStoreCache] + +PRs have also been opened for: + +- [Document supported file formats for dask_image.imread](https://github.com/dask/dask-image/issues/407) +- [Document supported file formats for skimage.io](https://github.com/scikit-image/scikit-image/issues/7879) + ## Improvements to cloud visualisation ## Acknowledgements From 6480a652cd3ab956bb0fc9502e5f18b5be8e888f Mon Sep 17 00:00:00 2001 From: ruaridhg Date: Tue, 23 Sep 2025 10:36:12 +0100 Subject: [PATCH 4/5] StoreLike Docs PR merged --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 6192977..cf52469 100644 --- a/README.md +++ b/README.md @@ -43,11 +43,11 @@ Contributions have been made to the zarr-python repository: - [Add CLI for converting v2 metadata to v3](https://github.com/zarr-developers/zarr-python/pull/3257) - [Added ArrayNotFoundError](https://github.com/zarr-developers/zarr-python/pull/3367) +- [Better document acceptable values for StoreLike](https://github.com/zarr-developers/zarr-python/pull/3480) PRs have been opened in the zarr-python repository: - [Prevent creation of arrays/groups under a parent array](https://github.com/zarr-developers/zarr-python/pull/3407) -- [Holding space - Better document acceptable values for StoreLike] - [Holding space - LRUStoreCache] PRs have also been opened for: From 1a0983b67c29c2dca5cee8bf23a7179501127a21 Mon Sep 17 00:00:00 2001 From: David Stansby Date: Thu, 9 Oct 2025 15:47:50 +0100 Subject: [PATCH 5/5] Apply suggestions from code review Co-authored-by: Kimberly Meechan <24316371+K-Meech@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index cf52469..4b94c2b 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ This textbook gives scientists: ## Benchmarking for Zarr -We created a set of benchmarks for writing data to Zarr with a range of different configurations as guidance for the options available when reading and writing 3D imaging data. +We created a [set of benchmarks](https://github.com/HEFTIEProject/zarr-benchmarks) for reading / writing data to Zarr with a range of different configurations. These benchmarks provide guidance on how selection of different configurations affect data size and read/write performance. The different parameters were: - Type of image