add gt obs getter for benchmark metrics#30
Conversation
|
what is this for? |
|
|
i mean more like, why do we care about this functionality? |
Oh, I see - I want to have first objective metrics in the benchmark around command tracking and this allows to get them through kos-sim in a clean way. |
|
But can't we just log the commands client-side? Why do we need to get them from kos? |
|
In order to get metrics we need to get ground truth information from the simulator which client does not have access to. |
|
oh, i see. basically you want a quantitative policy evaluation i'm not sure this is so useful, for the kscale-humanoid-benchmark - what i care about is the policy running on the real robot, not in simulation. not sure if these metrics will tell us anything that useful about whether or not it will work better on the real robot... |
I want to add first objective metrics in the benchmark around command tracking and this allows to get them through kos-sim in a clean way.
parallel pr in kos kscalelabs/kos#58

To plot/metric in benchmark deploy: