Hi Chen!
I am trying to reproduce the MaskedMimic full-body tracking results reported in
Table 1 of the MaskedMimic paper using the provided SMPL checkpoint:
data/pretrained_models/masked_mimic/smpl/last.ckpt
For MaskedMimic, Table 1 reports:
Split Success MPJPE
━━━━━━━━━━━━━ ━━━━━━━━━ ━━━━━━━━━
AMASS Train 99.4% 32.9 mm
───────────── ───────── ─────────
AMASS Test 99.2% 35.1 mm
However, the evaluation results I obtain are much lower than these reported
numbers.
My understanding of the evaluation protocol
Based on Section 7.2 and Section 8.1 of the paper, my understanding is:
-
The deployable MaskedMimic prior should be evaluated, rather than the
privileged posterior/encoder path.
-
The model should use the mean of the prior latent distribution. Therefore,
latent noise should be set to zero.
-
For full-body tracking, all features of the future target poses should be
visible to the model.
-
Evaluation should be performed on flat terrain.
-
A motion is considered failed if, at any frame, the average global joint
position deviation exceeds 0.5 m.
-
MPJPE should be reported in millimetres over global joint positions.
Please correct me if any part of this interpretation is incorrect.
Command used
Due to the currently available configuration options on this branch, I used:
python protomotions/inference_agent.py \
--checkpoint data/pretrained_models/masked_mimic/smpl/last.ckpt \
--simulator isaaclab \
--num-envs 256 \
--motion-file data/amass/amass_smpl_test.pt \
--full-eval \
--headless \
--overrides \
agent.model.vae.vae_noise_type=zeros \
agent.evaluator.evaluation_components.gt_error.static_params.threshold=0.50 \
agent.evaluator.save_predicted_motion_lib_every=None
My results are:
============================================================
EVALUATION RESULTS
============================================================
eval/action_delta_max_deg: 4.955446
eval/action_delta_max_rad: 0.086489
eval/action_delta_mean_deg: 0.837668
eval/action_delta_mean_rad: 0.014620
eval/action_rate_mean_rad_s: 0.438602
eval/gr_error/max: 3.021279
eval/gr_error/mean: 0.670585
eval/gr_error/min: 0.030782
eval/gt_error/failure_rate: 0.673913
eval/gt_error/max: 4.850602
eval/gt_error/mean: 0.368640
eval/gt_error/min: 0.006625
eval/high_jerk_frame_percentage_mean: 26.752044
eval/max_joint_error/max: 5.211261
eval/max_joint_error/mean: 0.679094
eval/max_joint_error/min: 0.016715
eval/normalized_jerk_mean: 1286.396362
eval/num_evaluated: 138.000000
eval/success_rate: 0.326087
Items Evaluated: 138
============================================================
Overall Score: 0.326087
============================================================
These results appear to be far from the 99.2% test success rate reported in
Table 1.
Full-conditioning and posterior diagnostics
As an additional diagnostic, I evaluated both the prior and the privileged encoder-decoder/posterior policy under full conditioning. In both cases, all available future joint information was provided without masking, and the latent noise was set to zero.
Prior with full future joint information
When the prior was given the full future joint information without any masking, it achieved a success rate of only 38.41%:
============================================================
EVALUATION RESULTS
============================================================
eval/gr_error/max: 3.016995
eval/gr_error/mean: 0.587996
eval/gr_error/min: 0.031202
eval/gt_error/failure_rate: 0.615942
eval/gt_error/max: 4.020061
eval/gt_error/mean: 0.294978
eval/gt_error/min: 0.006706
eval/max_joint_error/max: 4.543253
eval/max_joint_error/mean: 0.559957
eval/max_joint_error/min: 0.016551
eval/num_evaluated: 138.000000
eval/success_rate: 0.384058
Items Evaluated: 138
============================================================
Overall Score: 0.384058
============================================================
Encoder-decoder/posterior with full future joint information
When the privileged encoder-decoder/posterior branch was evaluated using the full future joint information without any masking, it achieved a success rate of only 43.48%:
============================================================
EVALUATION RESULTS
============================================================
eval/gr_error/max: 3.068513
eval/gr_error/mean: 0.567266
eval/gr_error/min: 0.030418
eval/gt_error/failure_rate: 0.565217
eval/gt_error/max: 3.617427
eval/gt_error/mean: 0.289864
eval/gt_error/min: 0.006679
eval/max_joint_error/max: 3.932021
eval/max_joint_error/mean: 0.550061
eval/max_joint_error/min: 0.016800
eval/num_evaluated: 138.000000
eval/success_rate: 0.434783
Items Evaluated: 138
============================================================
Overall Score: 0.434783
============================================================
I understand that the privileged encoder-decoder/posterior policy is not necessarily equivalent to the paper’s independently trained fully constrained controller (FC). However, since the encoder branch receives the complete dense mimic target, I would still expect it to achieve a success rate above 90%, or at least perform substantially better than the prior.
The posterior branch improves the success rate only slightly, from 38.41% to 43.48%, and both results remain far below the values reported in Table 1. This makes me wonder whether there may be a mismatch in the checkpoint, motion data preprocessing, observation construction, target representation, or evaluation configuration.
Questions
-
Could you please clarify why the provided MaskedMimic prior checkpoint achieves only a 38.41% success rate under full conditioning, whereas Table 1 reports 99.2%?
-
How should the similar performance of the prior and posterior be interpreted? In my evaluation, the posterior improves the success rate only from 38.41% to 43.48%, despite receiving the full dense target. I would appreciate any guidance on whether this behavior is expected or whether it may indicate a mismatch in the checkpoint, data processing, or evaluation configuration.
Hi Chen!
I am trying to reproduce the MaskedMimic full-body tracking results reported in
Table 1 of the MaskedMimic paper using the provided SMPL checkpoint:
However, the evaluation results I obtain are much lower than these reported
numbers.
My understanding of the evaluation protocol
Based on Section 7.2 and Section 8.1 of the paper, my understanding is:
The deployable MaskedMimic prior should be evaluated, rather than the
privileged posterior/encoder path.
The model should use the mean of the prior latent distribution. Therefore,
latent noise should be set to zero.
For full-body tracking, all features of the future target poses should be
visible to the model.
Evaluation should be performed on flat terrain.
A motion is considered failed if, at any frame, the average global joint
position deviation exceeds 0.5 m.
MPJPE should be reported in millimetres over global joint positions.
Please correct me if any part of this interpretation is incorrect.
Command used
Due to the currently available configuration options on this branch, I used:
My results are:
These results appear to be far from the 99.2% test success rate reported in
Table 1.
Full-conditioning and posterior diagnostics
As an additional diagnostic, I evaluated both the prior and the privileged encoder-decoder/posterior policy under full conditioning. In both cases, all available future joint information was provided without masking, and the latent noise was set to zero.
Prior with full future joint information
When the prior was given the full future joint information without any masking, it achieved a success rate of only 38.41%:
Encoder-decoder/posterior with full future joint information
When the privileged encoder-decoder/posterior branch was evaluated using the full future joint information without any masking, it achieved a success rate of only 43.48%:
I understand that the privileged encoder-decoder/posterior policy is not necessarily equivalent to the paper’s independently trained fully constrained controller (FC). However, since the encoder branch receives the complete dense mimic target, I would still expect it to achieve a success rate above 90%, or at least perform substantially better than the prior.
The posterior branch improves the success rate only slightly, from 38.41% to 43.48%, and both results remain far below the values reported in Table 1. This makes me wonder whether there may be a mismatch in the checkpoint, motion data preprocessing, observation construction, target representation, or evaluation configuration.
Questions
Could you please clarify why the provided MaskedMimic prior checkpoint achieves only a 38.41% success rate under full conditioning, whereas Table 1 reports 99.2%?
How should the similar performance of the prior and posterior be interpreted? In my evaluation, the posterior improves the success rate only from 38.41% to 43.48%, despite receiving the full dense target. I would appreciate any guidance on whether this behavior is expected or whether it may indicate a mismatch in the checkpoint, data processing, or evaluation configuration.