Skip to content

add gt obs getter for benchmark metrics#30

Open
budzianowski wants to merge 1 commit into
masterfrom
add_gt_obs_getter
Open

add gt obs getter for benchmark metrics#30
budzianowski wants to merge 1 commit into
masterfrom
add_gt_obs_getter

Conversation

@budzianowski

@budzianowski budzianowski commented May 1, 2025

Copy link
Copy Markdown
Contributor

I want to add first objective metrics in the benchmark around command tracking and this allows to get them through kos-sim in a clean way.

parallel pr in kos kscalelabs/kos#58
To plot/metric in benchmark deploy:
Screenshot 2025-04-30 at 19 01 38

@budzianowski budzianowski requested review from WT-MM and codekansas May 1, 2025 01:41
@CLAassistant

CLAassistant commented May 1, 2025

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@codekansas

Copy link
Copy Markdown
Member

what is this for?

@budzianowski

budzianowski commented May 1, 2025

Copy link
Copy Markdown
Contributor Author

what is this for?

kscalelabs/ksim-gym#12

@codekansas

Copy link
Copy Markdown
Member

i mean more like, why do we care about this functionality?

@budzianowski

budzianowski commented May 1, 2025

Copy link
Copy Markdown
Contributor Author

i mean more like, why do we care about this functionality?

Oh, I see - I want to have first objective metrics in the benchmark around command tracking and this allows to get them through kos-sim in a clean way.

@codekansas

Copy link
Copy Markdown
Member

But can't we just log the commands client-side? Why do we need to get them from kos?

@budzianowski

budzianowski commented May 2, 2025

Copy link
Copy Markdown
Contributor Author

In order to get metrics we need to get ground truth information from the simulator which client does not have access to.

@codekansas

Copy link
Copy Markdown
Member

oh, i see. basically you want a quantitative policy evaluation

i'm not sure this is so useful, for the kscale-humanoid-benchmark - what i care about is the policy running on the real robot, not in simulation. not sure if these metrics will tell us anything that useful about whether or not it will work better on the real robot...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants