The value of loss stays high from beginning to end within training time

[snailchan]

The value of loss stays high from beginning to end within training time
I’m sorry to disturb you again. For academic study, I planned to recurrent your experiment result. Thus I written the network file “matchnet_siamese.prototxt” and the solver file “matchnet_siamese_solver.prototxt”, according to the file you shared. However, limited by my ability, I just trained the network without pipelines you introduced.
But, when I supervised the output window, the value of loss stayed high, vibrating around 0.69, much higher than which I obtained when training other classifier network.
So I want to please you help me check where the error comes from.
Thanks for your reading.

matchnet_siamese.txt
matchnet_siamese_solver.txt

[killerjian007]

@snailchan , I also have the same problem, have you solved it ?

[snailchan]

@killerjian007 , I stopped here for a long time. If you had a solution, I would appreciate your sharing!

[pribadihcr]

Hi @snailchan @killerjian007 ,

I got the following error:
I1220 17:46:56.192490 14214 layer_factory.hpp:77] Creating layer pair_data
I1220 17:46:56.195097 14214 net.cpp:106] Creating Layer pair_data
I1220 17:46:56.195261 14214 net.cpp:411] pair_data -> pair_data
I1220 17:46:56.195397 14214 net.cpp:411] pair_data -> sim
I1220 17:46:56.312755 14220 db_leveldb.cpp:18] Opened leveldb /home/rudy/matchnet/data/leveldb/liberty.leveldb
I1220 17:46:56.340694 14214 data_layer.cpp:41] output data size: 64,1,64,64
I1220 17:46:56.347340 14214 net.cpp:150] Setting up pair_data
I1220 17:46:56.347489 14214 net.cpp:157] Top shape: 64 1 64 64 (262144)
I1220 17:46:56.347564 14214 net.cpp:157] Top shape: 64 (64)
I1220 17:46:56.347625 14214 net.cpp:165] Memory required for data: 1048832
I1220 17:46:56.347702 14214 layer_factory.hpp:77] Creating layer slice_pair
I1220 17:46:56.347810 14214 net.cpp:106] Creating Layer slice_pair
I1220 17:46:56.347884 14214 net.cpp:454] slice_pair <- pair_data
I1220 17:46:56.347990 14214 net.cpp:411] slice_pair -> data
I1220 17:46:56.348096 14214 net.cpp:411] slice_pair -> data_p
F1220 17:46:56.348245 14214 slice_layer.cpp:44] Check failed: top.size() <= bottom_slice_axis (2 vs. 1)
*** Check failure stack trace: ***

I used @snailchan matchnet_siamese & solver.prototxt above.

For your help, thank you very much.

[killerjian007]

Hi @pribadihcr
Your input data with only 1 channel?If so, you should generate 2 channel input data, that 1ch for an image, 2ch for another matched image, like caffe/examples/siamese.

[pribadihcr]

@killerjian007
I generated input data using default matchnet code: generate_patch_db.py.
do, I need generate data using convert_mnist_siamese_data in caffe ?.

[snailchan]

Hi, @killerjian007 , have you solved the problem?

[zhaishengfu]

i have the same problem also. i used my own data to train. maybe the format of training data is wrong??(i am trying to find the error).do you preprocess the training data as the paper??

[LeonSCZ]

@snailchan ,i've been on this experiment for a long long time,and my loss was still about 0.69,how can i know the way the author train this network

[zhengxiawu]

@LeonSCZ i had used matconvnet to do the network ,and found that the learning rate should be less than 0.00005, you can try it in caffe

[moustaphakaraki]

@zhengxiawu can you please share your solver parameters or loss function? Is the LR the only thing you changed? He used SoftmaxWithLoss here, but actually the paper talkes about Softmax+CrossEntropy, which isn't available in Caffe, since there is only SigmoidCrossEntropy. No idea if that is the problem though. I have the same 0.69 loss.

UPDATE: It turns out it had to do more with the weight initialization than the LR. If u try out some different fillers, it'll work. In my case, it worked with using guassian fillers with 0.1 std on the first conv layers.

[mrzw]

@mkaraki48 Do you have solve the problem?

[mrzw]

@pribadihcr How did you solve the error?

[Codersadis]

hi,
after training,
how can i convert the caffemodel/solverstate file to .pb file for pycaffe invoking?
is there any tools for the convertion from solverstate to .pb?

[niloup]

@mkaraki48 Could you please share your solver.prototxt that you have used to get reasonable response from training the network? I still get the 0.69 loss the entire training time even with the parameters that you have shared in your update. Thanks a lot!

[mayanksingh1998]

Can you please tell me how to train this model on my dataset