Problem
matnwb can create and export a DynamicTable containing a multidimensional VectorData backed by DataPipe whose stored row count does not match the table height. The resulting file exports successfully in MatNWB but cannot be read by PyNWB.
How to Reproduce
file = NwbFile( ...
"session_start_time", datetime("2021-01-01 00:00:00", "TimeZone", "local"), ...
"identifier", "ident1", ...
"session_description", "ExpandableTableTutorial" ...
);
start_time = types.hdmf_common.VectorData( ...
"description", "start times column", ...
"data", types.untyped.DataPipe( ...
"data", [1, 2], ...
"maxSize", Inf ...
) ...
);
stop_time = types.hdmf_common.VectorData( ...
"description", "stop times column", ...
"data", types.untyped.DataPipe( ...
"data", [2, 3], ...
"maxSize", Inf ...
) ...
);
ids = types.hdmf_common.ElementIdentifiers( ...
"data", types.untyped.DataPipe( ...
"data", int32([0; 1]), ...
"maxSize", Inf ...
) ...
);
file.intervals_trials = types.core.TimeIntervals( ...
"description", "test expandable dynamic table", ...
"colnames", {'start_time', 'stop_time'}, ...
"start_time", start_time, ...
"stop_time", stop_time, ...
"id", ids ...
);
vd = types.hdmf_common.VectorData( ...
"description", "data column with size 2 x 5", ...
"data", types.untyped.DataPipe( ...
"data", ones(2,5), ...
"maxSize", [Inf,5], ...
"axis", 1 ...
) ...
);
file.intervals_trials.addColumn("data", vd)
nwbExport(file, "expandableTableTestFile2.nwb");
Observed behavior
MatNWB accepts the new column and exports the file.
PyNWB then fails to read it with:
Could not construct TimeIntervals object due to: Columns must be the same length
In this repro, MatNWB validates vd as height 2, but the exported dataset is stored with shape 5 x 2, so PyNWB interprets the column as having 5 rows.
Expected behavior
DynamicTable.addColumn / dynamic table validation should reject this column before export because the stored row dimension does not match the table height.
Root cause
From local tracing in MatNWB:
- dynamic table validation uses
DataPipe.axis to determine DataPipe column height
- for multidimensional
DataPipe data, NWB/HDF5 storage order means the table row dimension is effectively the last MATLAB dimension on disk
- this lets a column pass MatNWB validation while still being invalid for downstream readers like PyNWB
So the issue is specifically a mismatch between:
- how MatNWB computes
DynamicTable height for multidimensional DataPipe columns
- how the exported dataset shape is interpreted by PyNWB
Problem
matnwbcan create and export aDynamicTablecontaining a multidimensionalVectorDatabacked byDataPipewhose stored row count does not match the table height. The resulting file exports successfully in MatNWB but cannot be read by PyNWB.How to Reproduce
Observed behavior
MatNWB accepts the new column and exports the file.
PyNWB then fails to read it with:
In this repro, MatNWB validates
vdas height2, but the exported dataset is stored with shape5 x 2, so PyNWB interprets the column as having5rows.Expected behavior
DynamicTable.addColumn/ dynamic table validation should reject this column before export because the stored row dimension does not match the table height.Root cause
From local tracing in MatNWB:
DataPipe.axisto determineDataPipecolumn heightDataPipedata, NWB/HDF5 storage order means the table row dimension is effectively the last MATLAB dimension on diskSo the issue is specifically a mismatch between:
DynamicTableheight for multidimensionalDataPipecolumns