Skip to content

DynamicTable.addColumn accepts multidimensional DataPipe column with mismatched stored row count #808

@ehennestad

Description

@ehennestad

Problem

matnwb can create and export a DynamicTable containing a multidimensional VectorData backed by DataPipe whose stored row count does not match the table height. The resulting file exports successfully in MatNWB but cannot be read by PyNWB.

How to Reproduce

file = NwbFile( ...
    "session_start_time", datetime("2021-01-01 00:00:00", "TimeZone", "local"), ...
    "identifier", "ident1", ...
    "session_description", "ExpandableTableTutorial" ...
);

start_time = types.hdmf_common.VectorData( ...
    "description", "start times column", ...
    "data", types.untyped.DataPipe( ...
        "data", [1, 2], ...
        "maxSize", Inf ...
    ) ...
);

stop_time = types.hdmf_common.VectorData( ...
    "description", "stop times column", ...
    "data", types.untyped.DataPipe( ...
        "data", [2, 3], ...
        "maxSize", Inf ...
    ) ...
);

ids = types.hdmf_common.ElementIdentifiers( ...
    "data", types.untyped.DataPipe( ...
        "data", int32([0; 1]), ...
        "maxSize", Inf ...
    ) ...
);

file.intervals_trials = types.core.TimeIntervals( ...
    "description", "test expandable dynamic table", ...
    "colnames", {'start_time', 'stop_time'}, ...
    "start_time", start_time, ...
    "stop_time", stop_time, ...
    "id", ids ...
);

vd = types.hdmf_common.VectorData( ...
    "description", "data column with size 2 x 5", ...
    "data", types.untyped.DataPipe( ...
        "data", ones(2,5), ...
        "maxSize", [Inf,5], ...
        "axis", 1 ...
    ) ...
);

file.intervals_trials.addColumn("data", vd)
nwbExport(file, "expandableTableTestFile2.nwb");

Observed behavior

MatNWB accepts the new column and exports the file.

PyNWB then fails to read it with:

Could not construct TimeIntervals object due to: Columns must be the same length

In this repro, MatNWB validates vd as height 2, but the exported dataset is stored with shape 5 x 2, so PyNWB interprets the column as having 5 rows.

Expected behavior

DynamicTable.addColumn / dynamic table validation should reject this column before export because the stored row dimension does not match the table height.

Root cause

From local tracing in MatNWB:

  • dynamic table validation uses DataPipe.axis to determine DataPipe column height
  • for multidimensional DataPipe data, NWB/HDF5 storage order means the table row dimension is effectively the last MATLAB dimension on disk
  • this lets a column pass MatNWB validation while still being invalid for downstream readers like PyNWB

So the issue is specifically a mismatch between:

  • how MatNWB computes DynamicTable height for multidimensional DataPipe columns
  • how the exported dataset shape is interpreted by PyNWB

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions