Skip to content

generalize the JSON serialization of scalars in data type definitions #351

Description

@d-v-b

Each array data type is associated with a scalar type. Those scalar values, i.e. the individual elements we write to arrays, have two representations described by the Zarr specs:

  • as elements of arrays, stored in chunks as encoded bytes.
  • as values associated with the fill_value key in metadata, in which case the value must be composed of JSON types.

There are many cases where users need to include scalars in array metadata. For example, if a codec takes an array of 64-bit floats as part of its configuration, or if a user wants to store an affine transformation in the attributes field, they would need a routine for storing these values in metadata.

Zarr itself has this problem with the fill_value field, which it solves in a very narrow fashion by defining a "fill value encoding" for each scalar type. But the specs could more generally define a JSON encoding for each scalar type, and then state that, e.g., "the fill_value field must be a scalar consistent with the array data type encoding uses the JSON scalar encoding defined with that data type".

Of course users today can just use language like "value X is saved in Zarr metadata using the fill value encoding for that data type", but I think it would be valuable to people if we re-arranged the language so that the fill value encoding was described as an application of a more generic JSON encoding for scalars.

As a bonus, if we formally define a JSON encoding for scalar values, it's also easy to define a JSON encoding for n-dimensional arrays of scalar values, by arranging JSON-encoded scalars into appropriately structured JSON arrays. But I don't think we need to include this language in the spec.

so: I'm proposing that we change the currentl language about fill_value encoding to instead describe JSON encoding for scalars, which is then applied to the fill_value field.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions