Feature: Write provenance to zarr from parent file + run details by BrianWhitneyAI · Pull Request #63 · bioio-devs/bioio-conversion

BrianWhitneyAI · 2026-07-15T19:44:08Z

Closes #64
Closes #65
Closes #66

Summary

Adds an opt-in include_provenance flag (default off) to OmeZarrConverter that captures where a converted store came from. When enabled, each scene's store gets:

A "bioio" attribute block in the root zarr.json with: source file name, UTC conversion timestamp, reader plugin, bioio package versions, StandardMetadata, and pointers to the metadata sidecars.
JSON sidecars at the store root: metadata.native.json (source-native metadata) and metadata.ome.json (normalized OME metadata), deduped to a single file when byte-identical (e.g. OME-TIFF).

Reader-specific kwargs can be forwarded to the provenance metadata reader via provenance_reader_kwargs — for example, to enable format-specific metadata extraction without affecting pixel reads.

Example output

my_image.ome.zarr/
├── zarr.json                  ← root attrs include "bioio" block
├── metadata.native.json       ← source-native metadata
└── metadata.ome.json          ← normalized OME metadata (omitted when identical to native)

zarr.json attributes:

{
  "bioio": {
    "source_file": "my_image.czi",
    "converted": "2026-07-24T18:00:00+00:00",
    "plugin": "bioio-czi",
    "bioio_python_versions": {
      "bioio": "3.0.0",
      "bioio-czi": "2.8.0",
      "bioio-conversion": "0.5.0"
    },
    "source_metadata": "metadata.native.json",
    "ome_metadata": "metadata.ome.json",
    "standard_metadata": {
      "physical_size_x": 0.108,
      "physical_size_y": 0.108,
      "...": "..."
    }
  }
}

🤖 Generated with Claude Code

Add an include_provenance flag (default off) that writes a top-level "bioio" attribute block beside the NGFF "ome" block in each scene's store, plus the source metadata XML as sidecars under bioio/. The block records the source file name, UTC conversion timestamp, reader plugin, bioio package versions (including bioio-conversion), the cross-format StandardMetadata for that scene, and pointers to the native/OME XML sidecars. Provenance assembly lives in a new bioio_conversion/provenance.py (ProvenanceBuilder), kept out of the converter proper. It is built once per conversion and cached: the metadata reader and whole-file XML are scene-independent, so only StandardMetadata is re-read per scene. For formats that surface richer metadata under non-default reader options, provenance opens a dedicated metadata reader (pixel I/O is unaffected): CZI uses an aicspylibczi reader with subblock metadata (embedding <Subblocks> into the native XML and deriving extra StandardMetadata such as total_time_duration), and ND2 uses the standard 96-well plate geometry so each scene's stage position is assigned a well row/column. The dedicated reader is best-effort and falls back to the pixel reader if it can't open or its scene names diverge. Provenance is keyed by scene name, not index, so a different metadata backend can't misattach a scene. Add a "provenance" install extra (bioio-czi, bioio-nd2) and pin bioio-nd2 in the test extras. Tests in tests/converters/test_provenance.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

BrianWhitneyAI · 2026-07-15T20:00:16Z

Example Provenance Output

Store layout

<name>.ome.zarr/
├── zarr.json                    # root attrs: { "ome": {...}, "bioio": {...} }
├── metadata.native.json         # source native metadata as JSON
├── metadata.ome.json            # normalized OME metadata as JSON
├── 0/                           # pyramid level 0
│   └── ...
└── 1/                           # pyramid level 1
    └── ...

Source metadata sidecars

# CZI (native != OME): two distinct sidecars
metadata.native.json   # raw <ImageDocument> serialized to JSON
metadata.ome.json      # normalized OME metadata as JSON
    source_metadata -> "metadata.native.json"
    ome_metadata    -> "metadata.ome.json"

# OME-TIFF (native already OME, byte-identical): one file, both pointers share it
metadata.ome.json
    source_metadata -> "metadata.ome.json"
    ome_metadata    -> "metadata.ome.json"

Full `zarr.json` — CZI (`variable_per_scene_dims.czi`, scene 0)

czi.ome.zarr / zarr.json

{
  "attributes": {
    "ome": {
      "version": "0.5",
      "multiscales": [
        {
          "name": "czi",
          "axes": [
            {"name": "t", "type": "time"},
            {"name": "c", "type": "channel"},
            {"name": "z", "type": "space"},
            {"name": "y", "type": "space"},
            {"name": "x", "type": "space"}
          ],
          "datasets": [
            {
              "path": "0",
              "coordinateTransformations": [
                {"type": "scale", "scale": [1.0, 1.0, 2.23, 0.5416666666666666, 0.5416666666666666]}
              ]
            },
            {
              "path": "1",
              "coordinateTransformations": [
                {"type": "scale", "scale": [1.0, 1.0, 2.23, 1.0833333333333333, 1.0833333333333333]}
              ]
            }
          ]
        }
      ],
      "omero": {
        "version": "0.5",
        "channels": [
          {
            "color": "#FFFFFF",
            "coefficient": 1.0,
            "active": true,
            "label": "CMDRP",
            "window": {"min": 0, "max": 255, "start": 0, "end": 255},
            "family": "linear",
            "inverted": false
          }
        ]
      }
    },
    "bioio": {
      "source_file": "variable_per_scene_dims.czi",
      "converted": "2026-07-24T18:00:00.000000+00:00",
      "bioio_python_versions": {
        "bioio": "3.0.1",
        "bioio-base": "3.3.0",
        "bioio-ome-zarr": "3.6.0",
        "bioio-conversion": "2.1.1",
        "bioio-czi": "2.8.0"
      },
      "plugin": "bioio-czi",
      "source_metadata": "metadata.native.json",
      "ome_metadata": "metadata.ome.json",
      "standard_metadata": {
        "binning": "1x1",
        "column": "4",
        "dimensions_present": "TCZYX",
        "image_size_c": 1,
        "image_size_t": 2,
        "image_size_x": 1848,
        "image_size_y": 1248,
        "image_size_z": 2,
        "imaged_by": "sara.carlson",
        "imaging_datetime": "2020-01-18T00:16:31.246361+00:00",
        "objective": "10x/0.45Air",
        "pixel_size_x": 0.5416666666666666,
        "pixel_size_y": 0.5416666666666666,
        "pixel_size_z": 2.23,
        "position_index": 1,
        "row": "4",
        "timelapse": true,
        "timelapse_interval": 59.927,
        "total_time_duration": 59.927
      }
    }
  },
  "zarr_format": 3,
  "consolidated_metadata": null,
  "node_type": "group"
}

Full `zarr.json` — ND2 (`ND2_dims_p4z5t3c2y32x32.nd2`, scene 2)

nd2.ome.zarr / zarr.json

{
  "attributes": {
    "ome": {
      "version": "0.5",
      "multiscales": [
        {
          "name": "nd2",
          "axes": [
            {"name": "t", "type": "time"},
            {"name": "z", "type": "space"},
            {"name": "c", "type": "channel"},
            {"name": "y", "type": "space"},
            {"name": "x", "type": "space"}
          ],
          "datasets": [
            {
              "path": "0",
              "coordinateTransformations": [
                {"type": "scale", "scale": [1.0, 1.0, 1.0, 0.652452890023035, 0.652452890023035]}
              ]
            }
          ]
        }
      ],
      "omero": {
        "version": "0.5",
        "channels": [
          {
            "color": "#FFFFFF",
            "coefficient": 1.0,
            "active": true,
            "label": "Widefield Green",
            "window": {"min": 0, "max": 255, "start": 0, "end": 255},
            "family": "linear",
            "inverted": false
          },
          {
            "color": "#FFFFFF",
            "coefficient": 1.0,
            "active": true,
            "label": "Widefield Red",
            "window": {"min": 0, "max": 255, "start": 0, "end": 255},
            "family": "linear",
            "inverted": false
          }
        ]
      }
    },
    "bioio": {
      "source_file": "ND2_dims_p4z5t3c2y32x32.nd2",
      "converted": "2026-07-24T18:00:00.000000+00:00",
      "bioio_python_versions": {
        "bioio": "3.0.1",
        "bioio-base": "3.3.0",
        "bioio-ome-zarr": "3.6.0",
        "bioio-conversion": "2.1.1",
        "bioio-nd2": "1.7.0"
      },
      "plugin": "bioio-nd2",
      "source_metadata": "metadata.native.json",
      "ome_metadata": "metadata.ome.json",
      "standard_metadata": {
        "binning": "1x1",
        "column": "3",
        "dimensions_present": "TZCYX",
        "image_size_c": 2,
        "image_size_t": 3,
        "image_size_x": 32,
        "image_size_y": 32,
        "image_size_z": 5,
        "imaged_by": null,
        "imaging_datetime": "2021-09-28T06:41:27.544008-07:00",
        "objective": "10x/0.3",
        "pixel_size_x": 0.652452890023035,
        "pixel_size_y": 0.652452890023035,
        "pixel_size_z": 1.0,
        "position_index": null,
        "row": "E",
        "timelapse": true,
        "timelapse_interval": 20.995054,
        "total_time_duration": 41.990108
      }
    }
  },
  "zarr_format": 3,
  "consolidated_metadata": null,
  "node_type": "group"
}

SeanDuHare

need to finish, but here are some initial thoughts

SeanDuHare · 2026-07-15T20:08:37Z

+            ``"bioio"`` attributes block (a sibling of ``"ome"``): the source's
+            cross-format ``standard_metadata``, the reader plugin and package
+            versions, and the native/OME metadata XML as sidecars under
+            ``bioio/``. For CZI this also captures per-subblock metadata. Off by


CZI bit feels unnecessary to mention - there will likely be quirks with each reader

SeanDuHare · 2026-07-15T20:11:48Z

        dtype: Optional[Union[str, np.dtype]] = None,
        n_workers: Optional[int] = None,
        shard_limit_bytes: int = DEFAULT_SHARD_LIMIT_BYTES,
+        include_provenance: bool = False,


provenance is such an overloaded term especially in biology - i feel like maybe "include_source_metadata" might help?

We also include creation metadata so I dont think "include_source_metadata" would cover it, we could split to two flags?

SeanDuHare · 2026-07-15T20:14:36Z

+        self._metadata_xml_cache: Optional[Tuple[Optional[str], Optional[str]]] = None
+        self._whole_file_cache: Optional[Tuple[Dict[str, Any], Dict[str, str]]] = None
+
+    def _metadata_reader(self) -> BioImage:


Why does this not re-use the bioimage reference from the caller? I think you did it this way because there are specific plugin parameters you want to use but that feels like it would be hard to discover as a user. Instead, providing warnings when certain information is missing could be a way to inform the user of the best route - I'll think about this more

It is very hard to discover as a user. The real problem is that you need to be a super user to get well info out of CZIs or ND2s, you need to know that CZI needs aicspylibczi to get subblock data to get well and nd2 needs a plate definition. Then you also need to know that aicspylibczi doesnt have the same efficiency as pylibczirw. It is not an easy problem but a "metadata probe" that uses the optimal metadata read and a "pixel data probe" for optimal reads was the pattern I was thinking.

SeanDuHare · 2026-07-15T20:18:55Z

+        """
+        Whole-file ``(ome_xml, native_xml)`` for the source, serialized once.
+        """
+        if self._metadata_xml_cache is None:


How necessary is this, is it just used per scene? Hos big can this get? I'm curious if memory is more precious here than a single file access

otherwise we do it for every scene. Chris Fricks most recent had 47 scenes, Ive seen as high as 180

I also think the subblock collection is somewhat expensive

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Replace semicolons in f-strings in provenance.py to avoid flake8 E702 false positives - Fix mypy error: pass Path(out_path) to write_sidecars instead of str, which also fixes sidecar files not being written (str / str fails silently inside write_sidecars) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove _metadata_reader_kwargs and its hardcoded aicspylibczi/nd2 logic from ProvenanceBuilder. Callers now pass metadata_reader_kwargs directly. OmeZarrConverter exposes this as provenance_reader_kwargs. Tests updated to pass aicspylibczi kwargs explicitly for the subblock test. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Add xmltodict>=0.13.0 dependency - Replace raw XML sidecar files with proper JSON via xmltodict.parse(); _OME_FORCE_LIST ensures repeating OME elements (Channel, Image, Plane, etc.) are always arrays regardless of count - Rename path constants: NATIVE_XML_PATH/OME_XML_PATH -> NATIVE_METADATA_PATH/OME_METADATA_PATH (.json extensions) - Update tests: read sidecars as JSON directly, no ET parsing needed; check root dict key instead of element tag; navigate Subblocks by key Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Move sidecars from bioio/ subdirectory to the store root as metadata.native.json and metadata.ome.json - Replace xmltodict + _OME_FORCE_LIST for OME metadata with ome-types model_dump(mode='json') for cleaner, well-typed output - CZI native (ET.Element) still goes through xmltodict - Add provenance_reader_kwargs passthrough to OmeZarrConverter and ProvenanceBuilder, removing hardcoded aicspylibczi/PLATE_96 logic - Update test_metadata_xml_sidecars -> test_metadata_json_sidecars to assert on model_dump key structure and dedup behaviour Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

No longer needed now that plugin-specific kwargs are user-supplied via provenance_reader_kwargs instead of hardcoded in the package. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

No nd2 fixtures or test references exist — it was carried over from the old provenance extra. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

AcquisitionTime is not guaranteed to be present in the test fixture; assert all subblocks are dicts instead, which proves include_subblock_metadata produced structured per-subblock data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

BrianWhitneyAI and others added 2 commits July 15, 2026 12:43

drop comment

14dc617

SeanDuHare reviewed Jul 15, 2026

View reviewed changes

This was referenced Jul 15, 2026

Feature Request: Surface bioio prov standard metadata bioio-devs/bioio-ome-zarr#156

Open

feature: surface provenance standard_metadata from the bioio block bioio-devs/bioio-ome-zarr#155

Draft

BrianWhitneyAI changed the title ~~Add opt-in source provenance to OME-Zarr conversion~~ Feature: Write provenance to zarr from parent file + run details Jul 15, 2026

BrianWhitneyAI and others added 8 commits July 23, 2026 14:23

Merge branch 'main' into feature/conversion-provenance

ade2fd2

bump bioio-ome-zarr minimum to 3.6.0 to pick up attributes support

f97308d

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: remove provenance install extra

a959c43

No longer needed now that plugin-specific kwargs are user-supplied via provenance_reader_kwargs instead of hardcoded in the package. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: drop bioio-nd2 from test dependencies

5c7998f

No nd2 fixtures or test references exist — it was carried over from the old provenance extra. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

BrianWhitneyAI marked this pull request as ready for review July 23, 2026 22:54

BrianWhitneyAI requested a review from a team as a code owner July 23, 2026 22:54

BrianWhitneyAI requested review from SeanDuHare, aswallace, kmitcham, pgarrison, saeliddp, smishra3 and toloudis July 23, 2026 22:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature: Write provenance to zarr from parent file + run details#63

Feature: Write provenance to zarr from parent file + run details#63
BrianWhitneyAI wants to merge 11 commits into
mainfrom
feature/conversion-provenance

BrianWhitneyAI commented Jul 15, 2026 •

edited

Loading

Uh oh!

BrianWhitneyAI commented Jul 15, 2026 •

edited

Loading

Uh oh!

SeanDuHare left a comment

Uh oh!

SeanDuHare Jul 15, 2026

Uh oh!

SeanDuHare Jul 15, 2026

Uh oh!

BrianWhitneyAI Jul 15, 2026

Uh oh!

SeanDuHare Jul 15, 2026

Uh oh!

BrianWhitneyAI Jul 15, 2026

Uh oh!

Uh oh!

SeanDuHare Jul 15, 2026

Uh oh!

BrianWhitneyAI Jul 15, 2026

Uh oh!

BrianWhitneyAI Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

BrianWhitneyAI commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Example output

Uh oh!

BrianWhitneyAI commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example Provenance Output

Store layout

Source metadata sidecars

Full zarr.json — CZI (variable_per_scene_dims.czi, scene 0)

Full zarr.json — ND2 (ND2_dims_p4z5t3c2y32x32.nd2, scene 2)

Uh oh!

SeanDuHare left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BrianWhitneyAI commented Jul 15, 2026 •

edited

Loading

BrianWhitneyAI commented Jul 15, 2026 •

edited

Loading

Full `zarr.json` — CZI (`variable_per_scene_dims.czi`, scene 0)

Full `zarr.json` — ND2 (`ND2_dims_p4z5t3c2y32x32.nd2`, scene 2)