Evaluation on PointOdyssey

[nikitakaraevv]

Hi @aharley,

Thank you for the great work!

I'm trying to reproduce the results on the test split of PointOdyssey. That's the link that I used to download the dataset. After following the installation instructions, I launched test_on_pod.py with S=128 and sur_thr=50, which produced the following output:

1_128_i16_pod05_212310; step 000001/12; rtime 0.97; itime 23.07; d_x 13.7; sur_x 23.0; med_x 69.7
1_128_i16_pod05_212310; step 000002/12; rtime 1.01; itime 26.43; d_x 16.9; sur_x 27.1; med_x 56.1
1_128_i16_pod05_212310; step 000003/12; rtime 13.41; itime 98.56; d_x 18.3; sur_x 35.4; med_x 45.0
1_128_i16_pod05_212310; step 000004/12; rtime 16.99; itime 134.77; d_x 28.2; sur_x 49.8; med_x 36.1
1_128_i16_pod05_212310; step 000006/12; rtime 19.53; itime 104.47; d_x 23.7; sur_x 40.7; med_x 47.3
1_128_i16_pod05_212310; step 000007/12; rtime 6.80; itime 42.10; d_x 24.4; sur_x 43.5; med_x 45.4
1_128_i16_pod05_212310; step 000008/12; rtime 3.20; itime 36.09; d_x 26.9; sur_x 48.2; med_x 42.0
1_128_i16_pod05_212310; step 000009/12; rtime 13.59; itime 121.77; d_x 25.3; sur_x 47.2; med_x 42.8
1_128_i16_pod05_212310; step 000010/12; rtime 13.38; itime 82.74; d_x 25.1; sur_x 48.6; med_x 50.3
1_128_i16_pod05_212310; step 000011/12; rtime 8.20; itime 75.24; d_x 28.2; sur_x 47.5; med_x 46.7
1_128_i16_pod05_212310; step 000012/12; rtime 10.96; itime 77.90; d_x 29.1; sur_x 47.0; med_x 44.8

My results are slightly different from what is described in the Testing section. Even though I have a different survival threshold, d_avg and median_l2 should be 31.3 and 33.0, respectively. Do you know why this might be the case?

In order to load the dataset, I had to change annotations.npz to annot.npz:

pips2/datasets/pointodysseydataset_fullseq.py

Line 62 in 5b24ec5

annotations_path = os.path.join(seq, 'annotations.npz')

and visibilities to visibs here:

pips2/datasets/pointodysseydataset_fullseq.py

Line 89 in 5b24ec5

visibs = annotations['visibilities'][full_idx].astype(np.float32)

Can it be a different version of the dataset?

[aharley]

Hi Nikita,
can you tell me the name and md5sum of the checkpoint you are running here, to make sure it's the one I used? Maybe I uploaded the wrong one.

./reference_model$: md5sum model-000200000.pth
e599fd3ba978d67c4cf35f225be3c2af  model-000200000.pth

[nikitakaraevv]

It looks like it's the same model:

reference_model$ md5sum model-000200000.pth 
e599fd3ba978d67c4cf35f225be3c2af  model-000200000.pth

My numbers on TAP-Vid match those in the Readme:
d_avg 70.5; survival_16 89.1; median_l2 6.9 compared to d_avg 70.6; survival_16 89.3; median_l2 6.9 in the Readme

[aharley]

Well that's good news at least. I guess the data changed, or I introduced a bug in the test script. If you are pushing for CVPR then I think the best move is just to evaluate things in apples-to-apples comparisons and not worry if it matches the readme/arxiv. If it's very valuable to you to get an exact match, let me know, and I can try to find some time and figure out what's wrong.

[nikitakaraevv]

Yes, we are pushing for CVPR. Okay, we will evaluate everything in an apples-to-apples comparison for now. Thank you for your help!