Bugs in computing the KL term in `SVGP`

[shixinxing]

Hi,

I found your work very interesting and helpful, but there seem to be two mistakes in lines 134-137 in SVGPVAE_model when you are computing the KL term of the lower bound $\mathcal{L}^l_H$ for the moving-ball experiment:

KL_term = 0.5*(K_mm_log_det - S_log_det - m +
                           tf.trace(tf.matmul(K_mm_inv, A_hat)) +
                           tf.reduce_sum(A_hat *
                                         tf.linalg.matvec(K_mm_inv, A_hat)))

1. When you compute the Mahalanobis distance, is A_hat supposed to be mu_hat? Should we also add axis=-1 in tf.reduce_sum?

2. Since you use tf.reduce_sum without the axis argument in lines 131-132, K_mm_log_det and S_log_det are two scalars. However, K_mm's shape is [M, M] whereas A_hat's [35, M, M] (M is the number of inducing points, 35 is the number of videos.). Therefore, we might need to retain the batch shape [35] for S_log_det; otherwise, we will miss a factor '35' before K_mm_log_det.

Could you please let me know if I am wrong? Thanks.