Evaluate proper home for Zarr bindings in kvikio

[vyasr]

Currently kvikio has some support for Zarr baked in. AFAICT (please correct me if I am wrong) kvikio is functioning as the Python bridge between Zarr in Python and underlying cufile (namely GDS) and nvcomp APIs. In the long run, we likely want nvcomp Python bindings to live in nvcomp itself rather than in kvikio (see #685), and we ought to consider the same for cufile. Zarr has already started moving in this direction with zarr-developers/zarr-python#2863 adding direct support for GPU zstd using nvcomp's own Python bindings. As such, we should consider whether in the long-term all the Zarr pieces of kvikio should eventually be migrated to be native in Zarr using other Python packages for cufile/nvcomp bindings.

[TomAugspurger]

AFAICT (please correct me if I am wrong) kvikio is functioning as the Python bridge between Zarr in Python and underlying cufile (namely GDS) and nvcomp APIs.

That's essentially correct. The slight caveat is around whether the user has zarr-python 2.x or 3.x installed. For zarr-python 3.x, the our store implementaiton really is just zarr's LocalStore with kvikio.CuFile in place of a regular python open().

we ought to consider the same for cufile

For my own understanding, is the rough idea that there'd be something like a nvidia.cufile.CuFile that provides what kvikio.CuFile currently provides? If that's the case, I suspect we could get the remainder of kvikio.zarr._zarr_python_3.GDSStore upstreamed to zarr-python.

[madsbk]

In any case, we still want a Zarr store (backend) that uses KvikIO as it provides good IO performance even when cuFile isn't available.

[jakirkham]

Given our past experiences adding CUDA support to third-party projects, think there remains value in having some portion of the code in RAPIDS for the following reasons:

1. Easy to fix on RAPIDS timeline
2. Able to leverage GPU CI
3. Minimizing burden on third-party projects

While it may make sense for some things to live in other libraries, remain convinced there is a need for a RAPIDS library

[vyasr]

For my own understanding, is the rough idea that there'd be something like a nvidia.cufile.CuFile that provides what kvikio.CuFile currently provides?

Yes, roughly. We've discussed having a proper long-term home for cufile bindings like this before (including @madsbk).

In any case, we still want a Zarr store (backend) that uses KvikIO as it provides good IO performance even when cuFile isn't available.

In a world where cufile and nvcomp each provide their own bindings separately, what would this code look like in kvikio? Would kvikio basically be a bridge between Zarr and the bindings from those packages?

While it may make sense for some things to live in other libraries, remain convinced there is a need for a RAPIDS library

Does it change much if much of the code is moved out of kvikio into other NVIDIA-managed non-RAPIDS packages (for nvcomp and cufile)?

[madsbk]

In a world where cufile and nvcomp each provide their own bindings separately, what would this code look like in kvikio? Would kvikio basically be a bridge between Zarr and the bindings from those packages?

It would be a Zarr store that uses kvikio.read/write and a Zarr codec that uses nvCOMP's official Python bindings.
I don't think we need a separate cuFile Zarr backend.

[ilan-gold]

Given our past experiences adding CUDA support to third-party projects, think there remains value in having some portion of the code in RAPIDS for the following reasons

I agree with this. I created https://github.com/ilan-gold/cuda-zarr with this intention (not saying I want to maintain it, but the time-to-something-that-works was much faster than waiting for zarr to merge) and I see zarr's modularity as a selling point. It lets you iterate - maybe nvcomp 4.X isn't perfect but you can iterate and give something to the community faster. There's no need to put things into zarr's core IMO, and you can then develop on your own timeline.

[vyasr]

That seems reasonable. I think we'll need to reevaluate exactly what the final state looks like once the nvcomp and cufile bindings are moved out of kvikio, but it sounds like we'll want to keep some part of the zarr plugin here.