save/load ML models

[SCiarella]

This closes #110, by adding a save/load utility for the ML modules using safetensors

Main changes:

• add safetensors as a dependency
• add diffwofost.ml_models.io with:
  • save_model(model, ...)
  • load_model(path, ...)
• store enough metadata to rebuild a saved model
• give models a default checkpoint name based on model structure
• add tests for save/load
• show how to use save/load in hybrid notebook
• add new function to API (docs)

Design choices:

• save_model(model) saves to a default temp location (ignored by git)
• the default filename depends on model class + init kwargs (deterministically)
• so the same model structure reuses the same checkpoint path
• saving again after retraining overwrites that file
• if you want multiple saved variants, you have to pass a custom filename

[SarahAlidoost]

@SCiarella thanks for implementing this 👍 . The io function looks great for saving and loading ML models! We want something similar for crop models and their parameters too. I noticed issue #110 didn't have much detail, so I added some explanations and examples to give a better idea of what I had in mind. The examples are just meant to explain the concept, the actual implementation can be different. We can also chat and brainstorm this offline!

[SCiarella]

Thanks @SarahAlidoost, I have totally missed that part!

In the new commit, I have added unified save_model and load_model functions to io.py that handle both ML models (via safetensors with state_dict() and constructor metadata) and physical models (Configuration + ParameterProvider, saved as a directory with config.json, parameters.safetensors, parameters.json, and any embedded CROP_NN_MODEL or CROP_COMPONENTS ML models).

I have also added some tests for the physical model io.

[SarahAlidoost]

@SCiarella thanks! look great! 🥇 I left a few comments to help improve readability; we can move some parts of save_model and load_model into config.py, but we don't need to add tests for those methods in test_config.py, the existing io tests should be enough.

I also noticed that io.py is currently under ml_models. Could please you move it one level up to src/diffwofost?

Other than that, this looks great and should be ready to merge.

[SarahAlidoost]

@SCiarella thanks for addressing the comments. Only one of the suggestions has not been addressed yet, see my comment. Do you think we can also implement the classmethod from_dict or is there any issue with that? thanks! After that, we can merge this PR.

[sonarqubecloud]

Quality Gate passed

Issues
4 New issues
0 Accepted issues

Measures
0 Security Hotspots
89.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud