Quick story of what I wanted to achieve:
As the training process takes quite some time ( CSR and CCIP models training combined will take around a week to train on my PC ).
So I analyzed sources a bit and found the parameter "save_every" in the autoencoder configuration class ConfigAE.
By default this parameter is set to 50. I have set it to 1, to make the model training checkpoint to be saved after each epoch - I noticed it is done in the directory "results/test/CSR/trained_models" :
So far so good. The goal was to be able to interrupt the process after any epoch and resume it at a later time.
But when I try to use the saved checkpoint file as a starting point, the saved epoch within it is not honored, and the training process starts again from epoch 1, even though the trainer script reports the CSR checkpoint is successfuly loaded:
Also the optimizer's and scheduler's state_dict is saved but not loaded.
Quick story of what I wanted to achieve:
As the training process takes quite some time ( CSR and CCIP models training combined will take around a week to train on my PC ).
So I analyzed sources a bit and found the parameter "save_every" in the autoencoder configuration class ConfigAE.
By default this parameter is set to 50. I have set it to 1, to make the model training checkpoint to be saved after each epoch - I noticed it is done in the directory "results/test/CSR/trained_models" :
So far so good. The goal was to be able to interrupt the process after any epoch and resume it at a later time.
But when I try to use the saved checkpoint file as a starting point, the saved epoch within it is not honored, and the training process starts again from epoch 1, even though the trainer script reports the CSR checkpoint is successfuly loaded:
Also the optimizer's and scheduler's state_dict is saved but not loaded.