Proposal: a multiply-to-unpack scale/offset codec

[TomNicholas]

tl;dr: The existing scale and offset codec is not suitable for VirtualiZarr

Summary

Propose a new sibling codec to scale_offset whose decode multiplies — decoded = stored * scale + offset — as opposed to scale_offset's decoded = stored / scale + offset. This is the convention used by CF conventions (NetCDF), GDAL/GeoTIFF, and ML quantization, and it lets those formats round-trip bit-for-bit, which scale_offset cannot.

Background

The recently-added scale_offset codec (zarr-developers/zarr-python#3874) decodes as stored / scale + offset and is the natural target for migrating CF-packed data (unpacked = packed * scale_factor + add_offset). In @maxrjones's review there, the conventions were tabulated, with the key note:

"CF's scale_factor=0.1 means the same thing as zarr's scale=10. Users migrating CF data need to invert the scale value."

That inversion is necessary but not sufficient for fidelity. Even with scale = 1/scale_factor, stored / scale ≠ stored * scale_factor in IEEE-754 for non-power-of-two factors, because the reciprocal isn't exact and division rounds differently than multiplication:

3 * 0.1   # 0.30000000000000004   (CF / decode_cf: multiply)
3 / 10.0  # 0.3                    (scale_offset: divide)

So a dataset migrated from CF to scale_offset no longer reproduces xarray.decode_cf exactly — decoded values shift by ~1 ULP. That's below the quantization floor packing already imposes, so it's not a correctness problem, but it does mean scale_offset cannot offer exact round-trip parity with the dominant geoscience decode convention, and tools that compare against canonical CF decoding see spurious diffs.

This came up while wiring scale_offset + cast_value into VirtualiZarr to represent CF scale_factor/add_offset packing as codecs (zarr-developers/VirtualiZarr#1004 / #1010). It works, but exact decode_cf parity is the one thing it can't deliver.

Why a new codec, not a flag on scale_offset

Codecs have no versioning or capability-negotiation mechanism — the name is the identity, and a codec is inherently must-understand. Adding e.g. {"configuration": {"convention": "cf"}} to scale_offset would change the semantics of an already-published codec; a reader that predates the flag has no version gate and would either error or silently ignore it and mis-decode (x / scale instead of x * scale) — reintroducing exactly the silent-corruption class this is meant to avoid. A distinct name makes the difference unmissable: a reader without the codec fails loudly rather than decoding wrongly.

Proposed codec

A array → array codec (same shape; intended to sit above cast_value, which handles the integer↔float cast), implementing the "multiply-to-unpack, scale-then-offset" convention:

• decode: out = in * scale + offset
• encode: out = (in - offset) / scale

Metadata:

{ "name": "<TBD>", "configuration": { "scale": 0.1, "offset": 5.0 } }

This single codec covers the families @maxrjones grouped together as "multiply to unpack, scale then offset": CF conventions (scale=scale_factor, offset=add_offset), GDAL/GeoTIFF, and ML quantization (real = (q - zero_point) * scale is the same family with offset = -zero_point*scale). HDF-EOS5 (scale*(stored - offset), offset-then-scale) and the existing scale_offset (multiply-to-pack) remain distinct.

cc @maxrjones @d-v-b — refs zarr-developers/zarr-python#3874, zarr-developers/VirtualiZarr#1004

[d-v-b]

{ "name": "cf.scale_offset", "configuration": { "scale": 0.1, "offset": 5.0 } }?

[d-v-b]

I'm fine with a second codec here. the spec can be short and define itself in reference to the existing scale_offset codec.

fwiw we did consider copying the cf scale-offset order of operations for scale_offset, but that would have hindered our second objective, which was similarity to the numcodecs scale-offset codec. We had to choose one.

[d-v-b]

{ "name": "cf.scale_offset", "configuration": { "divisive_scale": 0.1, "subtractive_offset": 5.0 } } if we want to be extra-explicit and ensure that this codec doesn't get confused with scale_offset