Debugging NeRF training on opengl-proc-gen data

[hayden-donnelly]

All training data is generated with opengl-proc-gen native (make run in submodules/opengl-proc-gen) and this data pack:
data.zip

With this commit:

2c939f802d1b7c3341f3e10b886f8da9c4883a74

I get these renders:

Training data for this had 60 views (x_steps = 2, y_steps = 30).

[hayden-donnelly]

There's some weird non-determinism with the training. Sometimes it will produce output like the above and sometimes it will produce full black or white images. Not sure why this is happening because it's being pointed to the same dataset. Maybe the data loader is modifying the dataset files when it loads them?

[hayden-donnelly]

I couldn't reproduce the results, then I tried spamming nvidia-smi during training and it started working again...

[hayden-donnelly]

Turning off high dynamic range seems to help. This makes the color activation a sigmoid instead of an exponential.