diff --git a/README.md b/README.md index f3f0b27..4b94c2b 100644 --- a/README.md +++ b/README.md @@ -13,8 +13,48 @@ This textbook gives scientists: - a guide to designing parallel processing algorithms to work efficiently with chunked datasets - a guide to exporting chunked datasets to other 'tradditional' datasets +## Benchmarking for Zarr + +We created a [set of benchmarks](https://github.com/HEFTIEProject/zarr-benchmarks) for reading / writing data to Zarr with a range of different configurations. These benchmarks provide guidance on how selection of different configurations affect data size and read/write performance. +The different parameters were: + +- Type of image + - Heart: HiP-CT scan of a heart from the Human Organ Atlas + - Dense: segmented neurons from electron microscopy + - Sparse: A few select segmented neurons from electron microscopy +- Software libraries + - Tensorstore (fastest for both reading and writing data) + - zarr-python version 3 + - zarr-python version 2 (slowest for both reading and writing data) +- Compressor + - blosc-zstd provides the best compression ratio, for image and segmentation data. (options were blosc-blosclz, blosc-lz4, blosc-lz4hc, blosc-zlib, blosc-zstd as well as gzip and zstd) +- Compression level + - Setting compression levels beyond ~3 results in slightly better data compression but much longer write times. Compression level does not affect read time. +- Shuffle + - Setting the shuffle option increases data compression with no adverse effect on read/write times (shuffle, bitshuffle and noshuffle were the 3 options) +- Zarr format version + - There was no noticeable difference between Zarr format 2 and Zarr format 3 data +- Chunk size + - Setting a low chunk size (below around 90) has an adverse effect on read and write times. + ## Tools for working with chunked datasets +Contributions have been made to the zarr-python repository: + +- [Add CLI for converting v2 metadata to v3](https://github.com/zarr-developers/zarr-python/pull/3257) +- [Added ArrayNotFoundError](https://github.com/zarr-developers/zarr-python/pull/3367) +- [Better document acceptable values for StoreLike](https://github.com/zarr-developers/zarr-python/pull/3480) + +PRs have been opened in the zarr-python repository: + +- [Prevent creation of arrays/groups under a parent array](https://github.com/zarr-developers/zarr-python/pull/3407) +- [Holding space - LRUStoreCache] + +PRs have also been opened for: + +- [Document supported file formats for dask_image.imread](https://github.com/dask/dask-image/issues/407) +- [Document supported file formats for skimage.io](https://github.com/scikit-image/scikit-image/issues/7879) + ## Improvements to cloud visualisation ## Acknowledgements