In sgkit-dev/vcztools#356 we added support for "proportional chunks" to vcztools. This was motivated by the unavoidable inefficiency of having millions of tiny chunks when the variants axis is large. The fix is to allow chunk sizes to vary over different arrays, but to try to keep chunks aligned across different arrays as much as possible by making them "proportional".
This will likely require some significant updates here, and we should think a bit about what some good heuristics would be. An early heuristic that we tried was to keep uncompressed chunks to roughly a given value (say 1Mb) but this doesn't work well as the chunk sizes can be quite different across arrays, and the GCD of chunk sizes across all arrays (what matters for vcztools) becomes quite small.
In sgkit-dev/vcztools#356 we added support for "proportional chunks" to vcztools. This was motivated by the unavoidable inefficiency of having millions of tiny chunks when the variants axis is large. The fix is to allow chunk sizes to vary over different arrays, but to try to keep chunks aligned across different arrays as much as possible by making them "proportional".
This will likely require some significant updates here, and we should think a bit about what some good heuristics would be. An early heuristic that we tried was to keep uncompressed chunks to roughly a given value (say 1Mb) but this doesn't work well as the chunk sizes can be quite different across arrays, and the GCD of chunk sizes across all arrays (what matters for vcztools) becomes quite small.