Skip to content

ADR-0007: Aperture BigQuery ↔ zarrmony field-mapping contract + instrument-metadata audit shape #60

Description

@ferrinm

What to build

Land ADR-0007 in docs/adr/ (and a companion reference at docs/references/aperture-bigquery-mapping.md) that:

  1. Documents the field-mapping contract with the Aperture BigQuery ingest team (schema in iac-aperture/deploy/arch/bigquery.tf). For every BigQuery column, states one of:

    • zarrmony-supplied — column source in the audit (path into attrs.zarrmony) and which reader paths populate it
    • user-supplied — the ingest form provides it, zarrmony does not attempt extraction
    • derived — the ingest layer computes it from a zarrmony field (e.g. uri_ome_zarr from the output path)
  2. Decides the audit shape for the new instrument-metadata block that follow-on slices populate. Open sub-decisions to resolve in the ADR body:

    • Per-scene vs file-level placement for acquisition_date, microscope (model), microscope_serial, imaging_method. Recommendation: per-scene consistently (values are simply duplicated across scenes when constant) so consumers only read from one place.
    • Shape of per_scene[i].channels — the identity block projecting existing extract_channels output for LIF and reader.ome_metadata.images[0].pixels.channels for CZI/ND2/OME-TIFF into one common surface (dye, fluor, excitation_nm, emission_low_nm, emission_high_nm).
    • Whether ome_ngff_version lands under config or a new top-level output block (folded in from the earlier standalone slice — trivial addition, but the ADR names its home).
  3. Publishes a per-format extractability matrix so downstream ingest knows what to expect. Rows: BigQuery columns marked zarrmony-supplied. Columns: LIF, CZI, ND2, default (OME-TIFF + other bioio plugins). Each cell: / / partial with a one-line note. Flags fields present in some readers but absent in others (e.g. objective in LIF today, absent in CZI/ND2/default until the follow-on slices land).

  4. Calls out channel_targets specifically — the ingest form is the primary source (RNAscope target genes, antibody names). Zarrmony populates it only when the source encodes probe/target names into channel names (occasional in Leica RNAscope acquisitions). The mapping doc flags this so ingest doesn't over-trust zarrmony's output.

Motivation

Another team is building the Aperture metadata parser/gate (iac-aperture) that will trigger data transfer, launch zarrmony conversion, and merge instrument metadata with user-supplied biological metadata in BigQuery. Today zarrmony's audit surfaces a fraction of what BigQuery wants; the rest sits in the input file and is discarded. Before writing per-format extractors we need a contract with the ingest team so both sides agree on shape, ownership, and per-format coverage.

Acceptance criteria

  • ADR-0007 committed in docs/adr/ following the existing ADR template style.
  • Reference mapping doc at docs/references/aperture-bigquery-mapping.md (or wherever fits the repo's docs convention) enumerating all BigQuery columns from iac-aperture/deploy/arch/bigquery.tf with status + source.
  • Per-format extractability matrix included.
  • Audit shape for per_scene[i].acquisition and per_scene[i].channels and file-vs-scene placement decisions locked.
  • ome_ngff_version home decided (folded in from earlier slice).
  • channel_targets provenance rules called out explicitly.
  • No code changes — this is documentation only.

Blocked by

None — can start immediately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentationenhancementNew feature or requestready-for-humanTriaged and ready for human attention/decision

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions