Crashes when testing

[askerlee]

When I did testing on VOC2012 using the pretrained resnet model, I encountered the following error:
seg_accuracy_layer.cpp:87] Unexpected label 220. num: 0. row: 109. col: 291
I tried to modify test.prototxt to ignore label 220, however I again encountered
seg_accuracy_layer.cpp:87] Unexpected label 38. num: 0. row: 109. col: 291
Seems I couldn't just keep adding all problematic labels. A Google search led to a suggestion to replace the "SegAccuracy" layer to an "Accuracy" layer. It indeed fixed the errors, however the final reported accuracy is
fc1_accuracy = 0.460454
which is far below the performance on the VOC leaderboard.
What's wrong here? Thanks for your help.

[dongzhuoyao]

I dont know why,but I'm sure you should not replace "SegAccuracy" to "Accuracy".just keep stayed.

[mjohn123]

Hi , I am also running the ResNet 101in VOC2012. It ran well in traning without error. However, when I start to run the testing, it has error about memory

I0101 00:03:58.796654 15999 caffe.cpp:252] Running for 1449 iterations.
F0101 00:03:59.323478 15999 syncedmem.cpp:56] Check failed: error == cudaSuccess (2 vs. 0)  out of memory
*** Check failure stack trace: ***
    @     0x7fdd0dd67daa  (unknown)
    @     0x7fdd0dd67ce4  (unknown)
    @     0x7fdd0dd676e6  (unknown)
    @     0x7fdd0dd6a687  (unknown)
    @     0x7fdd0e3cadb8  caffe::SyncedMemory::to_gpu()
    @     0x7fdd0e3c9e89  caffe::SyncedMemory::mutable_gpu_data()
    @     0x7fdd0e392f62  caffe::Blob<>::mutable_gpu_data()
    @     0x7fdd0e47c008  caffe::BaseConvolutionLayer<>::forward_gpu_gemm()
    @     0x7fdd0e59c5c6  caffe::ConvolutionLayer<>::Forward_gpu()
    @     0x7fdd0e3a4dc2  caffe::Net<>::ForwardFromTo()
    @     0x7fdd0e3a4ed7  caffe::Net<>::ForwardPrefilled()
    @           0x4074f7  test()
    @           0x405c88  main
    @     0x7fdd0d378f45  (unknown)
    @           0x406327  (unknown)
    @              (nil)  (unknown)
Aborted (core dumped)

How many memory do we need to run the testing model? I do not know why training can run but testing cannot run. I am using TitanX pascal 12GB

[xiaozai]

Hi,
I face the same problem. Can you tell me how did you solve it? Very appreciate.

F0109 15:25:10.716406 4907 seg_accuracy_layer.cpp:87] Unexpected label 220. num: 0. row: 148. col: 298
*** Check failure stack trace: ***

[dongzhuoyao]

your label is wrong.

[xiaozai]

@dongzhuoyao
hi,

I check the label ( the ground truth image provided by VOC2012) in matlab, there are only 0s, 1s, and 255s, and I already ignore the 255s. It seems that something wrong with the output of the Deeplab VGG network after fc8 layer.

Any suggestion?

[dongzhuoyao]

@xiaozai deeplabv1 or deeplabv2? if you want to try deeplabv1,you can use https://github.com/martinkersner/train-DeepLab,if you want to try deeplabv2,you can try the author's repository。

[xiaozai]

@dongzhuoyao

Thanks for this. I used the Deeplab v2, Because I want to get the accuracy with the test images. I input the test image into deploy net and add an seg_accuracy_layer to calculate the accuracy and check if it meets the leaderboard.

without the accuracy layer, everything is okay, the network passes and result got. with the accuracy layer, unexpected label error comes. My deploy net is same as the test net.

[sun11]

@xiaozai The reason is softmax loss function in caffe can only accept one channel ground truth labels, you have to convert the original indexed image in VOC12 SegmentationClass to single channel. This repo had already solved it in v1: https://github.com/martinkersner/train-DeepLab#data-conversions, it can also be applied to v2 exactly.

[xiaozai]

@sun11

Very thankful for this. I change the label to one channel image which only contains 0s and 1s with matlab and calculate the IoU with the matlab as well.

thanks for your help.

[dajiangxiaoyan]

@mjohn123 Do you fix the problem? I have met this crash when testing

[Jane-dudu]

@mjohn123 @dajiangxiaoyan Do you solve the "out of memory" problem when testing?

[Alinadi23]

@xiaozai I have the same problem with changing the masks to single channel.
would you please share your code to change masks?
thanks a lot

[xiaozai]

@Alinadi23 Hi, sorry I don't have the code anymore, but I remember the code is very easy to implement. Sorry for this.