Depth outputs all blank

[meder411]

I am running the code with the sample data that ships with the repo. Speed issues aside, all of the depth outputs I have seen so far (~25 frames) are all the same empty image. Each pixel in the output PNG is [68, 1, 84, 255]. Furthermore, I have to see the solver converge. Each frame runs 500 iterations and quits.

Am I doing something wrong? The intermediate images all look fine. The sparse depth images are a bit weak though. All I've done is run the code as is.

Here are the input, output and intermediate results for frame 39:

Input

Output

Image gradient

Flow

Flow gradient

Reliability

Soft edges

Solver initialization

Solver smoothness

Sparse points

[meder411]

After looking further into the problem, I haven't come up with a solution yet, but I've diagnosed the source. it appears the issue arises when applying the TemporalMedian function. For frame 14, the input and outputs are below.

INPUT

UNFILTERED DEPTH

FILTERED DEPTH (after applying TemporalMedian)

[holynski]

Thanks for bringing this to my attention. I'm looking into this now...

[holynski]

I'm not able to reproduce your error. Running the code as-is produces the following filtered depth map for frame 14:

Have you changed the code somehow in your version? If so, can you share the specific changes you've made?

[meder411]

Hm. Interesting. The only thing I did was to copy the code from an Jupyter notebook to a Python script. I am also running it in a conda environment too. Perhaps one of those two factors is playing a role? I will take a further look as well and report back too.

[holynski]

Are you able to produce the correct output from within the Jupyter notebook?

[meder411]

No, I can't. It must be something to do with the conda environment I guess? It runs with no errors, but the filtered depth is still empty for some reason (when running in a Jupyter notebook). I am using opencv 4.1, opencv-contrib-python 4.1.0.25, and pyquaternion 0.9.5, running on a Ubuntu 18.04 server with Python 3.7.3.

[holynski]

Here's my setup:

OSX 10.14.3
Python 3.7.2 (via conda)
OpenCV 4.0.0
SciPy 1.2.1
PyQuaternion 0.9.5
NumPy 1.15.4

To open and run the code, I start a Python kernel with jupyter notebook, and then navigate to the notebook, and press "Run All".

If indeed there is an issue within the TemporalMedian function, I'm not sure what would cause it, apart from a change in the np.median function. Could you tell me the size of the array (depth_maps) passed to the TemporalMedian function?

Just to double check -- you do mean frame 000014.png, and not the 14th frame which is saved, correct? This is first frame which is saved if skip_frames is set to 0.

Also, for sanity -- you haven't modified the COLMAP files, right?

[meder411]

I do mean frame00014 and I haven't modified the COLMAP files at all. It is odd though, because you're right, TemporalMedian is basically just a numpy function and that's a fairly boilerplate operation that's unlike to be the cause. Interestingly, the subsequent solver initialization images all look great. I'm thinking there's a visualization issue somewhere for me.

Solver initialization of frame000015.png after frame000014.png's output was blank

[holynski]

Ah, that's definitely possible. Let me look into that code.

[holynski]

Can you inspect the min/max values of the filtered depth map, and compare it to that of the solver initialization?

It could be an issue with the automatic scaling.

[meder411]

Hm, results below:

==>Processing frame000014.png
Initialization min: 0.5802239
Initialization max: 1.266863
Raw depth min: 0.80136484
Raw depth max: Inf
Filtered depth min: 0.80136484
Filtered depth max: Inf
==>Processing frame000015.png
Initialization min: 0.5802239
Initialization max: 1.266863
Raw depth min: 0.80864733
Raw depth max: Inf
Filtered depth min: 0.81061256
Filtered depth max: Inf

I believe you are right. It's the Inf's that are screwing it up it seems--throwing off the scale. I haven't found the specific source of those yet, but it's I'm fairly sure there's a divide-by-zero error somewhere.

[holynski]

Yep - very likely a divide-by-zero. I've actually seen debug warnings on my end for a while, but never had the time to investigate. Let's see if I can find it now.

[holynski]

Found it at the end of the DensifyFrame function, when copying the solver disparities back into the depth map.

            depth[col,row] = 1.0 / x[col + row * w]

Will push a fix shortly.