feat: support aligned writing on NetCDF->Zarr path

[LDeakin]

Used by default in the CLI when converting to Zarr.

Add input_aligned_chunks parameter to dataset_to_zarr and ignore_block_chunks to NetCDF reader, both defaulting to false. Used by default in the CLI on the NetCDF->Zarr path.

Ignoring block chunks is necessary because sometimes the internal chunking in our NetCDF datasets be irregularly gridded between blocks. With input_aligned_chunks, the Zarr output will just match the NetCDF blocking (but will use subchunking).

• add lint recipe
• Use chunks/subchunks internally in zarr/_writer.py. Makes the logic much clearer to me (and LLMs?)

Note

This is all severely overcomplicated by the fact that we are not utilising rectilinear grids yet (waiting on ecosystem support) and that the sharding codec requires subchunks to evenly divide the chunk shape. I am working on lifting that restriction in the spec. Also, I would ultimately prefer to go a virtualisation path in the future for NetCDF -> Zarr so that we can avoid rewriting any chunk data.

[Copilot]

Pull request overview

This PR enhances the NetCDF → Zarr conversion pipeline by adding an “input-aligned” writing mode for Zarr output and a NetCDF read option to ignore per-file internal chunking, aiming to preserve/align chunk grids and avoid expensive rechunking.

Changes:

• Add input_aligned_chunks to dataset_to_zarr() and refactor writer layout resolution to an internal chunks/subchunks model, with optional skip-rechunk behavior.
• Add ignore_block_chunks to NetCDF reading to optionally treat each NetCDF file/block as a single Dask chunk (useful for irregular internal NetCDF chunking across blocks).
• Update CLI to default to the new behavior (NetCDF→Zarr), add worker control and optional Dask graph visualization support; add a lint just recipe.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/anu_ctlab_io/zarr/_writer.py	Adds input_aligned_chunks, introduces chunks/subchunks internal layout, and conditionally skips rechunking before storing.
src/anu_ctlab_io/netcdf/__init__.py	Adds ignore_block_chunks and sets xarray chunking defaults to respect internal chunks unless overridden.
cli/src/anu_ctlab_io_cli/_cli.py	Uses ignore_block_chunks on NetCDF→Zarr, enables aligned writing for Zarr output, adds --workers and --dask-graph.
tests/test_zarr_write.py	Updates layout tests for the chunks/subchunks refactor and adds coverage for input-aligned writing behavior.
tests/test_netcdf.py	Adds tests validating default internal-chunk respect and the new “ignore block chunks” behavior.
cli/pyproject.toml	Adds graphviz dependency for Dask graph visualization.
uv.lock	Locks the new graphviz dependency.
justfile	Adds lint recipe (pre-commit run --all-files).
CHANGELOG.md	Documents the new input_aligned_chunks parameter.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[LDeakin]

Evaluation - $21624 \times 2914 \times 2914$ dataset

NetCDF style chunking (block: $32 \times 2914 \times 2914$) with anu_ctlab_io CLI:

• tomoHiRes_nc (676 files, 441GB) -> tomoHiRes.zarr (678 files, 191GB)
• ~12 minutes on 1 node = $2SU \times (12/60) \times 48 \times 1 =19.2 SU$

Using mango with default chunking: $512^3$ / subchunk: $32 ^3$. Need a lot of nodes due to memory usage.

• tomoHiRes_nc (676 files, 441GB) ->tomoHiRes_SS.zarr (1762 files, 229GB)
• 2:20 minutes on 12 nodes = $44.8 SU$

Note

• Mango is generating multiscale data + using blosclz level 9 with shuffling,
• anu_ctlab_io is single level + zstd level 0