Describe the bug
I am trying to run the NeRf implementation using the same code base. However, I am getting a memory over run issue. What should the configuration setting be?
To Reproduce
Steps to reproduce the behavior:
Configuration used for training:
export DATASET="/nitthilan/data/NSVF/Synthetic_NSVF/Spaceship/"
export SAVE="./spaceship_nerf_ckpt/"
export TRAIN_DIM="50x50"
export TRAIN_VIEWS="0..100"
export VALID_DIM="1x1"
export VALID_VIEWS="0..100"
export PRUNE_EVERY=2500
export VIEW_PER_BATCH=1
export PIXEL_PER_VIEW=2048
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -u train.py ${DATASET}
--user-dir fairnr
--task single_object_rendering
--train-views ${TRAIN_VIEWS} --view-resolution ${TRAIN_DIM}
--max-sentences 1 --view-per-batch ${VIEW_PER_BATCH} --pixel-per-view ${PIXEL_PER_VIEW}
--no-preload
--sampling-on-mask 1.0 --no-sampling-at-reader
--valid-views ${VALID_VIEWS} --valid-view-resolution ${VALID_DIM}
--valid-view-per-batch 1
--transparent-background "1.0,1.0,1.0" --background-stop-gradient
--arch nerf_base
--color-weight 128.0 --alpha-weight 1.0
--optimizer "adam" --adam-betas "(0.9, 0.999)"
--lr 0.001 --lr-scheduler "polynomial_decay" --total-num-update 150000
--criterion "srn_loss" --clip-norm 0.0
--num-workers 0
--seed 2
--save-interval-updates 500 --max-update 150000
--virtual-epoch-steps 500 --save-interval 1
--half-voxel-size-at "5000,25000,75000"
--reduce-step-size-at "5000,25000,75000"
--pruning-every-steps ${PRUNE_EVERY}
--keep-interval-updates 5 --keep-last-epochs 5
--log-format simple --log-interval 1
--save-dir ${SAVE}
--raymarching-tolerance 0.01
--tensorboard-logdir ${SAVE}/tensorboard
| tee -a $SAVE/train.log
Expected behavior
If I increase the TRAIN_DIM="50x50" to anything more then the memory consumed exceeds 25GB. I am running it on a 4 GPU system. Ideally, the same configuration runs for nsvf_base arch for dimensions of 800x800. Am I missing something?
Describe the bug
I am trying to run the NeRf implementation using the same code base. However, I am getting a memory over run issue. What should the configuration setting be?
To Reproduce
Steps to reproduce the behavior:
Configuration used for training:
export DATASET="/nitthilan/data/NSVF/Synthetic_NSVF/Spaceship/"
export SAVE="./spaceship_nerf_ckpt/"
export TRAIN_DIM="50x50"
export TRAIN_VIEWS="0..100"
export VALID_DIM="1x1"
export VALID_VIEWS="0..100"
export PRUNE_EVERY=2500
export VIEW_PER_BATCH=1
export PIXEL_PER_VIEW=2048
export CUDA_VISIBLE_DEVICES=0,1,2,3
python -u train.py ${DATASET}
--user-dir fairnr
--task single_object_rendering
--train-views ${TRAIN_VIEWS} --view-resolution ${TRAIN_DIM}
--max-sentences 1 --view-per-batch ${VIEW_PER_BATCH} --pixel-per-view ${PIXEL_PER_VIEW}
--no-preload
--sampling-on-mask 1.0 --no-sampling-at-reader
--valid-views ${VALID_VIEWS} --valid-view-resolution ${VALID_DIM}
--valid-view-per-batch 1
--transparent-background "1.0,1.0,1.0" --background-stop-gradient
--arch nerf_base
--color-weight 128.0 --alpha-weight 1.0
--optimizer "adam" --adam-betas "(0.9, 0.999)"
--lr 0.001 --lr-scheduler "polynomial_decay" --total-num-update 150000
--criterion "srn_loss" --clip-norm 0.0
--num-workers 0
--seed 2
--save-interval-updates 500 --max-update 150000
--virtual-epoch-steps 500 --save-interval 1
--half-voxel-size-at "5000,25000,75000"
--reduce-step-size-at "5000,25000,75000"
--pruning-every-steps ${PRUNE_EVERY}
--keep-interval-updates 5 --keep-last-epochs 5
--log-format simple --log-interval 1
--save-dir ${SAVE}
--raymarching-tolerance 0.01
--tensorboard-logdir ${SAVE}/tensorboard
| tee -a $SAVE/train.log
Expected behavior
If I increase the TRAIN_DIM="50x50" to anything more then the memory consumed exceeds 25GB. I am running it on a 4 GPU system. Ideally, the same configuration runs for nsvf_base arch for dimensions of 800x800. Am I missing something?