Set of dependencies/files to keep/delete

[fatwir]

Action item from June 1 Cyto-DL WGF

[pgarrison]

Specifically, we're looking for a proposed list of cyto-dl components that will be valuable to many future AICS ML projects and are general-purpose enough to be re-used without project-specific modifications.

[kiryteo]

I think @smishra3 has already worked on a base version for this at https://github.com/AllenCell/CytoDL-Mini, so we can start off that!

[smishra3]

Cyto-dl contents common to all projects -

1. Repo scaffolding

   • README.md
   • pyproject.toml
   • Makefile
   • LICENSE
   • version.toml
   • .bumpversion.cfg
   • .gitignore
   • .env.example
   • .pre-commit-config.yaml
   • requirements/{requirements,test-requirements,docs-requirements}.txt
   • data/.gitkeep, logs/.gitkeep (standard layout)

2. GitHub config

   • .github

3. Infra configs

   • configs/train.yaml
   • configs/eval.yaml
   • configs/compile.yaml
   • configs/hydra
   • configs/extras
   • configs/paths
   • configs/callbacks
   • configs/logger
   • configs/trainer
   • configs/debug
   • configs/hparams_search

4. Core Python packages

• Entry points

  • cyto_dl/init.py
  • cyto_dl/train.py
  • cyto_dl/eval.py
  • cyto_dl/compile.py

• Utils

  • cyto_dl/utils/init.py
  • cyto_dl/utils/arg_checking.py
  • cyto_dl/utils/array.py
  • cyto_dl/utils/checkpoint.py
  • cyto_dl/utils/config.py
  • cyto_dl/utils/download_test_data.py
  • cyto_dl/utils/dummy_dataset.py
  • cyto_dl/utils/pylogger.py
  • cyto_dl/utils/rich_utils.py
  • cyto_dl/utils/template_utils.py

• Callbacks

  • cyto_dl/callbacks/init.py
  • cyto_dl/callbacks/image_saver.py
  • cyto_dl/callbacks/layer_freeze.py
  • cyto_dl/callbacks/outlier_detection.py
  • cyto_dl/callbacks/tabular_saver.py

• Dataframe + datamodules (base infra)

  • cyto_dl/dataframe
  • cyto_dl/datamodules/init.py
  • cyto_dl/datamodules/array.py
  • cyto_dl/datamodules/data_dict.py
  • cyto_dl/datamodules/dataframe
  • cyto_dl/datamodules/folder.py
  • cyto_dl/datamodules/multidim_image.py
  • cyto_dl/datamodules/smartcache.py (??)
  • cyto_dl/datamodules/torchvision.py

• Image I/O + transforms (generic)

  • cyto_dl/image/init.py
  • cyto_dl/image/io/init.py
  • cyto_dl/image/io/bioio_loader.py
  • cyto_dl/image/io/monai_bio_reader.py
  • cyto_dl/image/io/numpy_reader.py
  • cyto_dl/image/io/ome_zarr_reader.py
  • cyto_dl/image/io/skimage_reader.py
  • cyto_dl/image/transforms/init.py
  • cyto_dl/image/transforms/add_meta.py
  • cyto_dl/image/transforms/bright_sampler.py
  • cyto_dl/image/transforms/clip.py
  • cyto_dl/image/transforms/contrastadjust.py
  • cyto_dl/image/transforms/coordinate_crop.py
  • cyto_dl/image/transforms/merge.py
  • cyto_dl/image/transforms/multiscale_cropper.py
  • cyto_dl/image/transforms/pad.py
  • cyto_dl/image/transforms/project.py
  • cyto_dl/image/transforms/save.py

• Loggers

  • cyto_dl/loggers/init.py
  • cyto_dl/loggers/mlflow.py

• Model base classes + framework

  • cyto_dl/models/init.py
  • cyto_dl/models/base_model.py
  • cyto_dl/models/basic_model.py
  • cyto_dl/models/im2im/init.py
  • cyto_dl/models/im2im/multi_task.py
  • cyto_dl/models/im2im/utils/init.py
  • cyto_dl/models/im2im/utils/inferers.py
  • cyto_dl/models/im2im/utils/noise_annealer.py
  • cyto_dl/models/im2im/utils/postprocessing/**
  • cyto_dl/models/handlers/** (base/image/load_image_patch handlers)
  • cyto_dl/models/utils/**

• NN base comp

  • cyto_dl/nn/init.py
  • cyto_dl/nn/head/init.py
  • cyto_dl/nn/head/base_head.py
  • cyto_dl/nn/losses/init.py
  • cyto_dl/nn/losses/loss_wrapper.py
  • cyto_dl/nn/mlp.py
  • cyto_dl/nn/res_unit.py
  • cyto_dl/nn/torchvision_wrapper.py

• API

  • cyto_dl/api/init.py
  • cyto_dl/api/data.py
  • cyto_dl/api/model.py
  • cyto_dl/api/cyto_dl_model/init.py
  • cyto_dl/api/cyto_dl_model/cyto_dl_base_model.py

5. Dependencies — keep in default pyproject.toml

   • hydra-core, hydra-colorlog, omegaconf, pyrootutils, PyYAML
   • torch, lightning, monai, einops
   • numpy, pandas, pyarrow, tqdm, rich, fire, matplotlib
   • mlflow
   • universal-pathlib
   • scikit-learn
   • bioio, bioio-base, bioio-tifffile, bioio-ome-tiff, tifffile, ome-zarr, bioio-ome-zarr

6. Tests

   • tests/init.py
   • tests/conftest.py
   • tests/helpers/**
   • tests/utils.py
   • tests/test_configs.py (validates default configs)
   • tests/test_train.py
   • tests/test_eval.py
   • tests/test_mlflow_logger.py
   • tests/test_array_models.py
   • tests/test_alternating_batch_sampler.py
   • tests/api/**
   • tests/resources/{rand_im.tif,train.csv,valid.csv,test.csv}
   • tests/torchvision_wrapper.py

[fatwir]

June 18 WGF meeting

Action Items

• Make a unified reader. There seems to be a lot of overlap between the existing readers!
• What is a good base config file that can be used?
• Future Task - Try out model architecture and hyperparameter search using all the available compute
• Check alternatives to OmegaConf/Hydra - maybe use pydantic
• Pair Programming with Philip/Jess with changing composability, understanding how the existing cyto-dl setup works

Notes

• multi_task.py under Model base classes + framework is very much needed, the other classes are maybe not as important.
• How much config file composability we'll use depends on the project stage, so we'll need to determine that.