Extensions for N5 support

[clbarnes]

I am hacking on N5 support for zarrs and it can pretty much all be decomposed into extensions.

May be of interest to @bogovicj @mkitti @d-v-b .

There's a MVP which would allow metadata-only conversions from N5 to Zarr - i.e. owners of N5 data would just stick a zarr.json file next to every attributes.json.
The stretch goal of needing no on-disk changes would require a storage transformer, which there's not a lot of prior art for, and I'm not sure it's viable in any case (see below).

Chunk key encoding

Needs to turn the request for chunk (z,y,x) into the path "x/y/z".

Chunk grid

N5 blocks each encode their own shape, with the idea being that blocks at the end of an array where the array shape isn't an integer multiple of the block shape can encode that they are effectively truncated.
Assuming that this is the only case where this truncation happens, we could define a chunk grid which encodes this expectation.
Zarrs already does this with the regular_bounded chunk grid - I'm not sure if there are equivalents elsewhere in the ecosystem.

Note that in zarr-python v2's N5 support (and tensorstore), the trailing chunks were padded rather than truncated.
Such data would probably cause the regular_bounded chunk grid to fail.

Codec

N5 blocks comprise a header followed by encoded data, which means the N5 codec must be an array-to-bytes codec which has compression configuration inside it, rather than as part of the outermost codec chain.

N5 block data is always see comment below C order (I think; not entirely clear), despite the blocks themselves being in F order (I think) , and always bigendian. So config like

{
  "node_type": "array",
  ...
  "codecs": [
    {
      "name": "n5",
      "configuration": {
        // N5's compression is effectively an optional bytes-to-bytes codec
        "bytes_to_bytes_codec": {
          "name": "gzip",
          "configuration": { "level": 6 }
        }
      }
    }
  ]
}

would result in a Zarr codec which, on read

1. reads the header
   • check for default/varlen/object modes
   • determine the size of the block (should be the chunk size, or smaller if it's on the end)
2. strips the header and applies a zarr codec chain for the described decompression, then bigendian byte ordering
3. possibly an array-to-array codec for padding the (truncated N5) end block shape to the (padded zarr) end chunk shape, as an alternative to the regular_bounded chunk grid above
   • possibly related to Request: Truncate codec #44

This should be the only codec.

Storage transformer?

My current implementation is a store which wraps any other zarr store and redirects zarr.json requests to attributes.json, parses the N5 metadata response and re-encodes it as Zarr metadata.
This seems like a job for a storage transformer.

However, we can only read a storage transformer configuration from a zarr.json, and if the target data has a zarr.json, then we don't need the storage transformer (as the N5 codec and chunk key encoding would both be in that Zarr metadata).
So it's not clear how useful this would actually be.
Instead, we could just describe the steps an N5StoreWrapper would need to take and leave it up to the implementors.

[d-v-b]

However, we can only read a storage transformer configuration from a zarr.json, and if the target data has a zarr.json, then we don't need the storage transformer (as the N5 codec and chunk key encoding would both be in that Zarr metadata).
So it's not clear how useful this would actually be.
Instead, we could just describe the steps an N5StoreWrapper would need to take and leave it up to the implementors.

this exposes a serious limitation in the storage transformer concept. Given that, the path you are taking (store logic) seems sensible. What probably needs to be specified is the transformation from a zarr.json file to an attributes.json file. We probably need a new category for store extensions to express this.

[mkitti]

N5 blocks each encode their own shape, with the idea being that blocks at the end of an array where the array shape isn't an integer multiple of the block shape can encode that they are effectively truncated.

The chunking scheme can be expressed as a subset of rectilinear chunking:

#25

In many cases, the chunk header can be ignored may be static. My plan has been to use the pad codec for this purpose:
#36

[clbarnes]

The chunking scheme can be expressed as a subset of rectilinear chunking:

Seems like a lot of metadata and implementation complexity (for readers) for a very tiny and simple case within the rectilinear scheme. I also don't know what the performance implications are for rectilinear chunking compared to regular or if the chunk size lookup cost gets completely washed away by the IO cost, like everything else.

I would prefer to avoid it if possible (even though Lachy is well ahead of us on implementing rectilinear chunking in zarrs!).

In many cases, the chunk header can be ignored may be static. My plan has been to use the pad codec for this purpose:

Yes, for a given array configuration we can make some assumptions about the header size and content, especially if we don't want to support varlen or object N5 arrays. In any case, it's not necessarily clear what we should do if the header doesn't meet our expectations, so it's just a question of whether we want to validate every chunk header or go ahead with the assumption (possibly leading to some hard-to-explain errors down the line).

[mkitti]

Needs to turn the request for chunk (z,y,x) into the path "x/y/z".

I would have thought that the transpose codec might have helped here.

[mkitti]

Seems like a lot of metadata and implementation complexity (for readers) for a very tiny and simple case within the rectilinear scheme. I also don't know what the performance implications are for rectilinear chunking compared to regular or if the chunk size lookup cost gets completely washed away by the IO cost, like everything else.

The run-length-encoding scheme makes the metadata quite compact and looks up relatively easy. I do not think IO will be an issue at all, especially if implementations use specialized code for this common case:

{
    // ...
    "shape": [1000, 1000],
    "chunk_grid": {
        "name": "rectilinear",
        "configuration": {
            "kind": "inline",
            "chunk_shapes": [
                [[64, 15], 40],
                [[32, 31], 8]
            ]
        }
    }
}

That said, I have argued that there should be a configuration scheme to succinctly describe boundary chunk truncation which does not require the metadata writer (possibly a human) to do math:

#25 (comment)

[clbarnes]

I would have thought that the transpose codec might have helped here.

I did start with transpose but I think it's not actually what we want - the transpose codec transposes the array represented by a single chunk, where we want to transpose the chunk grid, but not any of the data in the chunk.

This seems to work in my implementation (transposing the chunk index request and then not transposing the chunk data).

[jbms]

Note:

Let's say an N5 array has blockSize 100,200,300 and axes x,y,z.

Then the chunk with chunk coordinates cx,cy,cz is stored at cx/cy/cz and is stored in Fortran order, where x is the innermost dimension.

However the confusing part is that some implementations of N5, notably the old zarr-python n5store, decided to transpose their view of the array, and would report this array as having blockSize 300,200,100 with axes z,y,x and C order storage of the chunks.

I don't recommend doing that transpose by default since it causes all sorts of confusion.

[clbarnes]

Ah, so it only appeared to work in my unit tests because I generated the test data using the zarr N5 store... sigh. I'll go back to trying to install n5-utils.

In that case, we don't need a chunk key encoding, because we can use the v2 /-separated encoding; then we do need to use a transpose codec to read the F-order N5 chunks.

[mkitti]

TensorStore also has a N5 driver. If we need N5 support for Python, this is primarily what we're using internally at Janelia right now. Some do use zarr-python v2 still, but that's becoming unsustainable.

[mkitti]

We, the Janelia Data Standards group, recently just discussed label-multiset data that we typically store in N5. Essentially this format allows us to assign multiple labels to a single voxel.

@cmhulbert and I have started the process of generating documentation on the format, and we plan on proosing these as zarr-extensions.

[mkitti]

@clbarnes If you would like to pursue this, perhaps we should arrange a call. I will DM on imagesc Zulip.

[clbarnes]

It seems that tensorstore also pads the edge chunks when writing. But I've rebased my impl on test data generated by tensorstore for the transposition handling, and thrown in a couple more codecs. The fake metadata produced by the store wrapper looks something like

Details

// N5 attributes.json
{
  "blockSize":[256,128],
  "compression":{"level":0,"type":"zstd"},
  "dataType":"float32",
  "dimensions":[256,128]
}

// zarr zarr.json
{
  "zarr_format": 3,
  "node_type": "array",
  "shape": [
    256,
    128
  ],
  "data_type": "float32",
  "chunk_grid": {
    "name": "zarrs.regular_bounded",
    "configuration": {
      "chunk_shape": [
        256,
        128
      ]
    }
  },
  "chunk_key_encoding": {
    "name": "v2",
    "configuration": {
      "separator": "/"
    }
  },
  "fill_value": 0,
  "codecs": [
    {
      "name": "zarrs.n5",
      "configuration": {
        "codecs": [
          {
            "name": "transpose",
            "configuration": {
              "order": [
                1,
                0
              ]
            }
          },
          {
            "name": "bytes",
            "configuration": {
              "endian": "big"
            }
          },
          {
            "name": "zstd",
            "configuration": {
              "level": 0,
              "checksum": false
            }
          }
        ]
      }
    }
  ]
}

I suspect that some issues remain in mapping default compressor config values between Java and Rust, but as I'm only decoding that probably doesn't matter.

The transpose and bytes codec config doesn't necessarily need to be made explicit in the fake metadata serialization because they must always be there for the N5 codec, but it would make things more consistent if it ever became possible to represent N5 as "just zarr".

[mkitti]

If I recall, there are is still an unresolved issue with the LZ4 compressor:
zarr-developers/numcodecs#175

[clbarnes]

I suspect there are N5s in the wild which are written by implementations which either do or don't pad the end chunks, which means any reader will have to understand the N5 block header to figure out whether there is padding. I believe this means we will always need an N5-specific codec, rather than being able to just ignore the header and use a pad codec. That codec needs to read the block header before the bytes-to-bytes codecs, and then reshape/ trim/ pad the array according to that header after the array-to-bytes step, which means it must take the form I've used here (an array-to-bytes codec which wraps an optional compressor, a bigendian Bytes, and a Transpose).

[cmhulbert]

In an effort to support Zarr3 via the N5 API, we have (api internally) migrated to a similar codec-style for handling data (and metadata) in N5. While we have no intention of changing the N5 file-format, we are still intending to continue developing N5 API as a data access layer, especially for the java ecosystem.

That said, we have "codec" style implementations on n5 master branch that should give some idea of how we do block parsing, it may be useful as a reference if you implement something similar. Happy to talk more in call as well. Here's the main entry point for what we refer to as N5BlockCodecs and here is the relevant n5-zarr codec that pads the blocks

There are currently n5 alpha releases that use this, but a stable release is still a work in progress (hopefully soon)