While running the CCIP trainer, first 5 epochs finish successfully, but when the validation step kicks in, I get the subject exception, as detailed below:
Traceback (most recent call last): File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/train_gencad.py", line 187, in <module> train_model(args.model, args) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/train_gencad.py", line 108, in train_model ccip_trainer.train(train_loader=train_loader, val_loader=val_loader, ckpt=args.ckpt_path) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/trainer/ccip_trainer.py", line 164, in train val_loss = self.val_one_epoch(val_loader) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/trainer/ccip_trainer.py", line 77, in val_one_epoch loss = self.model(batch_cad, batch_image, return_loss=True, freeze_cad_encoder=True) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/ccip_model.py", line 315, in forward enc_image = self.image_encoder(image) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 272, in forward z = self.encoder(x) # (B, 512, 8, 8) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 221, in forward return self._forward_impl(x) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 216, in _forward_impl x = self.layer4(x) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/container.py", line 240, in forward input = module(input) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 93, in forward out = self.bn1(out) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/batchnorm.py", line 193, in forward return F.batch_norm( File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/functional.py", line 2822, in batch_norm return torch.batch_norm( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 3.94 GiB of which 19.75 MiB is free. Including non-PyTorch memory, this process has 3.91 GiB memory in use. Of the allocated memory 3.76 GiB is allocated by PyTorch, and 79.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
The validation dataset is supposed to be much smaller than the training dataset.
So, I guess at the time of validation the GPU still contains the training dataset.
Anyway, will debug it further.
PS: having this error on NVIDIA GeForce GTX 1050 Ti GPU with 4 Gb of RAM.
While running the CCIP trainer, first 5 epochs finish successfully, but when the validation step kicks in, I get the subject exception, as detailed below:
Traceback (most recent call last): File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/train_gencad.py", line 187, in <module> train_model(args.model, args) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/train_gencad.py", line 108, in train_model ccip_trainer.train(train_loader=train_loader, val_loader=val_loader, ckpt=args.ckpt_path) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/trainer/ccip_trainer.py", line 164, in train val_loss = self.val_one_epoch(val_loader) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/trainer/ccip_trainer.py", line 77, in val_one_epoch loss = self.model(batch_cad, batch_image, return_loss=True, freeze_cad_encoder=True) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/ccip_model.py", line 315, in forward enc_image = self.image_encoder(image) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 272, in forward z = self.encoder(x) # (B, 512, 8, 8) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 221, in forward return self._forward_impl(x) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 216, in _forward_impl x = self.layer4(x) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/container.py", line 240, in forward input = module(input) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/media/collabgear/SSD-M2-1TB/PycharmProjects/GenCAD/GenCAD/model/image_encoder.py", line 93, in forward out = self.bn1(out) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/modules/batchnorm.py", line 193, in forward return F.batch_norm( File "/home/collabgear/miniconda3/envs/gencad_env/lib/python3.10/site-packages/torch/nn/functional.py", line 2822, in batch_norm return torch.batch_norm( torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 3.94 GiB of which 19.75 MiB is free. Including non-PyTorch memory, this process has 3.91 GiB memory in use. Of the allocated memory 3.76 GiB is allocated by PyTorch, and 79.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)The validation dataset is supposed to be much smaller than the training dataset.
So, I guess at the time of validation the GPU still contains the training dataset.
Anyway, will debug it further.
PS: having this error on NVIDIA GeForce GTX 1050 Ti GPU with 4 Gb of RAM.