The value of learning_rate.momentum

[huangzehao]

Hi, minhnhat93. I am confused about the value of learning_rate.momentum in your code lista_train.
Would you mind telling me how to choose the value? Thank you very much!

[minhnhat93]

Hi, are you working with Homa?
The code on github is very out-dated. The current code is also a big mess.
I'm working on it now.
The learning_rate.momentum I believe is "to adds a fraction m of the
previous weight update to the current one". Value for
learning_rate.momentum should be 0.1 to 0.9. My English is bad but you can
search gradient descent with momentum on the internet. For example this is
a good read:
http://ufldl.stanford.edu/tutorial/supervised/OptimizationStochasticGradientDescent/.
This learning_rate.momentum in the code is correspond to γ in the article.

Minh Nhat Nguyen
Student ID: 51102403
Department of Computer Science and Engineering
Ho Chi Minh University of Technology (Vietnam National University)
Vietnam
Mobile: (84) 966-739-841
Email: minhnhat93@gmail.com

On Fri, Oct 23, 2015 at 4:39 PM, FlyingKid notifications@github.com wrote:

Hi, minhnhat93. I am confused about the value of learning_rate.momentum in
your code lista_train.
Would you mind telling me how to choose the value? Thank you very much!

—
Reply to this email directly or view it on GitHub
#1.

[huangzehao]

Thank you very much. Your answer help me!
And I am not working with Homa. I just want to do an implement of this paper "Deep Networks for Image Super-Resolution with Sparse Prior" in iccv 2015.
It's based on LISTA and the author didn't share his training code.
So I am trying to implement LISTA firstly.
I have readed your code and fixed some simple bug.
I test it but I found that "lista_test_script.m" takes so much time. Is this test function work?

You are still working this code now? I think it can be simpply implement by some deep learning frameworks,such as caffe, cuda-convnet and so on. Do you have any idea about this?

I am a Chinese student, my English is also bad. But it's nice to communicate with you, a helpful guy, thank you !

[minhnhat93]

Hi,
You are correct. LISTA is just a recurrent neural network unfolded. When I
first started with this project I didn't know about much about those deep
learning frameworks and the algorithm in the paper were clear therefore I
just went ahead and implemented it from scratch. You can use other
frameworks to train this I think. The activation function in this LISTA is
the soft threshold function is pretty uncommon thus I dont know if it's
available in those frameworks. If you can embed the soft thresholding
function in those frameworks, LISTA just become a normal feed-forward
network with shared parameters, which mean the only difference is you have
to take sum of gradients for those parameters, everything else is the same.
That lista_test_script I wrote for so long therefore I not totally sure if
it's working or not. I'm doing my thesis now so I dont have much time to
spent on developing the code anymore except documenting but I will take a
look. On the first thought I think that it is working but you have to use a
machine with NVIDIA CPU and the code is making it pause after running each
test image so you may want to check that too.
Best

Minh Nhat Nguyen
Student ID: 51102403
Department of Computer Science and Engineering
Ho Chi Minh University of Technology (Vietnam National University)
Vietnam
Mobile: (84) 966-739-841
Email: minhnhat93@gmail.com

On Fri, Oct 23, 2015 at 10:24 PM, FlyingKid notifications@github.com
wrote:

Thank you very much. Your answer help me!
And I am not working with Homa. I just want to do an implement of this
paper "Deep Networks for Image Super-Resolution with Sparse Prior" in iccv
2015.
It's based on LISTA and the author didn't share his training code.
So I am trying to implement LISTA firstly.
I have readed your code and fixed some simple bug.
I test it but I found that "lista_test_script.m" takes so much time. Is
this test function work?

You are still working this code now? I think it can be simpply implement
by some deep learning frameworks,such as caffe, cuda-convnet and so on. Do
you have any idea about this?

I am a Chinese student, my English is also bad. But it's nice to
communicate with you, a helpful guy, thank you !

—
Reply to this email directly or view it on GitHub
#1 (comment).