PathSGD optimization

[dewenzeng]

Hi Behnam,

Very interesting work. I just have a quick question, do the self.ratio and path_params (scaling factor for each parameter) in path_sgd.py need to be re-calculated according to the latest model updated in the latest mini-batch? or are they calculated based on the initialized model and remain constant? To me, it seems that they are calculated in the very first mini-batch and then remain unchanged.

[DCalhas]

At first, it seemed to me that it is only computed in the first mini-batch. Then, the paper refers to the neural balance theorem. And by my understanding, what you're doing is getting the "geometry" of the network by calculating the path-norms. If you use the update rule proposed (with these path norms), the neural network becomes balanced. Such that $w^0 \sim w^t$. So it appears that you could compute the path-norms at every update, but since the proposed update assures neural balance, you don't need to do it for all the batches.

Also, the complexity of the algorithm is $\mathcal{O}((B+1)T)$. The constant 1, in my interpretation, refers to the computation of the path-norms. In this case, this tells me you compute the path-norms once for every batch.

But as I said, due to it ensuring neural balance, it can be computed only once. For the first update of the learning session.

I am writing this also to check if I am correct. Hope it helps.

EDIT: I kept repeating the computation of the path norms, but I was referring to the ratio. The path norms are computed for every batch.