Coping with custom data of arbitary shape

[Mohinta2892]

Hello!

I am wondering if your bs prepare works when there are multiple input files in the same root dir?
Does it create several round_{num} directories per dataset?

Instead, if I want one model to ingest all the input files - what should I do differently?
Also, are there any data preprocessing stages in prepare such that every dataset has the same size/shape/normalisation?

Best,
Samia

[yajivunev]

Hello!

By input files do you mean volumes (zarr containers with at least a raw_dataset)? If yes, bs prepare first prepares the volumes (dumped to round_{num}/volumes.yaml), which you can prepare multiple of (by specifying the number of volumes when prompted) in a round. It does not create round directories for each volume. In a round, all volumes that have a labels_dataset will be used for ingestion (training), and all volumes (since all must have an raw_dataset) are target volumes for inference and post-processing.

You can also prepare volumes ahead of preparing configs with bs prepare volumes and loading the dumped yaml when preparing round configs.

Datasets can have arbitrary size/shape. Normalization is not included in prepare at the moment and would have to be checked manually / adjusted as needed in the gunpowder pipelines (in root/round_dir/setup_dir/{train,predict}.py)

prepare does have some general prompts while preparing volumes (such as asking if you want to do bounding box / create a mask) -- the bs utils commands also has clahe normalization at the moment. bs utils convert (which is called when preparing volumes if input file is .tif or a directory of .tifs) handles some basic normalization.

Let me know if that works -- I will be revamping the README, examples, and documentation in the coming days !

Thank you for your interest!
Best,
Vijay

[Mohinta2892]

Many thanks Vijay!

Understood, I am going to give this a spin.
My files are .hdf volumes, which I can convert to zarrs.

From your comments, it seems that the input data requirement is either a .zarr or .tif.
Are there jpeg/png plugins at all? I have another dataset which currently contains png/jpeg files split into images and labels directories. Any tips on ingesting these besides data format conversion ?

Best,
Samia

[yajivunev]

I plan to include support for HDFlike arrays. For now, I just pushed commit 7f8ae26 which removes the .tif conditions since tifffile.imread can read more than just .tifs (not yet tested on non-tifs) -- let me know how it works for your other dataset!

uint8 for storing image arrays is advisable (especially if the volumes can get large). uint32 or uint64 for labels. If you can view your zarrs arrays in neuroglancer, and if your zarr arrays have "offset", "voxel_size", "axis_names", and "units" as attributes -- then I would expect things to run smoothly : )

[Mohinta2892]

Many thanks Vijay! I shall check and report back if I find issues.