Issues with IoU Computation in The Eval Code

[voidrank]

WSIS_BBTP/matlab/EvalVOCInstSeg.m

Line 75 in 03cb87b

Overlap(j,i) = sum(TempProposal & TempGTInst) / sum(TempProposal | TempGTInst);

The code computes the intersection over union (IoU) between the ground-truth mask and the predicted mask.

There are two severe issues in the above line.

1. Both TempProposal and TempGTInst are matrices (2D tensors), sum(TempProposal & TempGTInst) and sum( TempProposal | TempGTInst) are vectors (1D tensors), which represent the intersection and union of each column of segmentation images. In Matlab, if we use the operation / to connect 2 vectors (let's say vector a divided by vector b), it will output a scalar r, such that |a - b * r| is minimal. However, r is used as IoU by mistake. I didn't see any relation between r and IoU and I check several r values during evaluation, none of them is equals to the correct IoU values.

2. When debugging this line, I found that Intersection eliminates those ignored pixels but Union does not.

[chengchunhsu]

Hi @voidrank,

Thanks for pointing it out.
We have been aware of this problem and will release an errata soon.

For now, you can fix this by changing the line
Overlap(j,i) = sum(TempProposal & TempGTInst) / sum(TempProposal | TempGTInst);
as
Overlap(j,i) = sum(sum(TempProposal & TempGTInst)) / sum(sum(TempProposal | TempGTInst));

Cheng-Chun

[voidrank]

Hi @chengchunhsu

Thanks for the confirmation.

Will be great if you can also pay attention to the second issue I mentioned. There may be another bug of how ignored pixels are handled in computing IoU.

[voidrank]

To be more specific, the ignored pixels means those edge pixels marked as 255th category, which should be ignored in computing IoU. However, I found that the evaluation code doesn't consider ignored pixels in computing intersection but takes ignored pixels into account when computing union.

[chengchunhsu]

Sure. I will get back to you after I detect the problem.
Thanks for the effort.

Cheng-Chun