Hello! Thank you so much for this beautiful paper; I have really enjoyed learning from the insights of both its mathematical and experimental demonstrations :)
I am having trouble reproducing the generalization results of the paper, which I had been very excited about as they are at the heart of this algorithm. I wanted to ask your advice about what I am doing wrong?
In walker _ walk, I tested several different circumstances, by changing the img_source for the env and the img_source for the eval_env. I tested
img_source for env / img_source for eval_env:
noise / color
color / noise
noise / none
to examine whether the perofrmance gneralizes or not. I left all other hyperparameters as default in your code.
In walker_walk, in almost all cases above (and for 3 seeds), when bisim_coef=0.5 --> training reward ~ 90, and in eval: reward ~70
and when bisim_coef=0.0 --> training reward ~ 50, and in eval: reward ~30
Does this match what you would expect, and am I doing the generalization experiments correctly? (I was unable to find how you did the training on simple distractors and evaluated on natural video from the code base, so I did these ones).
Thanks so much in advance for the kind help and happy 2023! :)
Hello! Thank you so much for this beautiful paper; I have really enjoyed learning from the insights of both its mathematical and experimental demonstrations :)
I am having trouble reproducing the generalization results of the paper, which I had been very excited about as they are at the heart of this algorithm. I wanted to ask your advice about what I am doing wrong?
In walker _ walk, I tested several different circumstances, by changing the img_source for the env and the img_source for the eval_env. I tested
img_source for env / img_source for eval_env:
noise / color
color / noise
noise / none
to examine whether the perofrmance gneralizes or not. I left all other hyperparameters as default in your code.
In walker_walk, in almost all cases above (and for 3 seeds), when bisim_coef=0.5 --> training reward ~ 90, and in eval: reward ~70
and when bisim_coef=0.0 --> training reward ~ 50, and in eval: reward ~30
Does this match what you would expect, and am I doing the generalization experiments correctly? (I was unable to find how you did the training on simple distractors and evaluated on natural video from the code base, so I did these ones).
Thanks so much in advance for the kind help and happy 2023! :)