Question: Any plan for third-stage RL fine-tuning in MaskedMimic for improved generalization?

[csyufei]

Thanks for open-sourcing this amazing project, the modular design and the MaskedMimic framework are incredibly inspiring.

I have a question regarding the training pipeline of MaskedMimic.

As I understand, the current pipeline consists of two stages:

1. Training a full-body expert tracker (via RL)
2. Distilling it into a masked policy (MaskedMimic)

However, in my experiments (specifically in VR tracking controller settings, where the tracked joints are fixed to the head and two wrists, or even just the two wrists), I found that while the second-stage policy performs well on in-distribution tasks, it tends to struggle with unseen or out-of-distribution goals. The generalization ability in these sparse-tracking scenarios seems somewhat limited.

This made me wonder: Is there any plan to introduce a third-stage RL fine-tuning phase (similar to PULSE), where the masked policy is further optimized with reinforcement learning?

Thanks again for the great work, looking forward to your insights!

Best,
Yufei Zhu

[tesslerc]

This is currently an interesting and open question.
https://arxiv.org/abs/2503.22886 this provides an example for how to fine-tune. While the performance is promising, the motion quality can use some improvement.
We're looking into better methods. Once someone (us or external) achieves high quality fine-tuning results we'll integrate directly into the main repo.

Our goal is for the repo to be maintained and hold methods that we find successful and that can withstand the test of time.

[tesslerc]

We have now released an update to ProtoMotions to support model fine-tuning.

On the original question:

Third-stage RL fine-tuning (GPC) is a reasonable direction, particularly when specializing MaskedMimic to a fixed sparse-tracking setup such as head-and-wrists or wrists-only control.

Our recent update adds shared infrastructure for loading pretrained modules and fine-tuning them with PPO. It also introduces SFT, RLFT, optional AMP rewards, LoRA/DoRA adapters, and frozen-reference constraints for GPC.

These additions make a MaskedMimic RL fine-tuning stage much easier to implement, but the released GPC PEFT code is not directly interchangeable with MaskedMimic.

GPC operates on discrete autoregressive tokens, while MaskedMimic currently uses a continuous VAE prior and action decoder (https://github.com/NVlabs/ProtoMotions/blob/main/protomotions/agents/supervised/masked_mimic_model.py). A MaskedMimic implementation would still need a small policy wrapper that adds a PPO action distribution and critic. It could then either fine-tune the full deployable policy or attach adapters, while using a frozen reference/KL constraint or AMP reward to limit degradation in motion quality.

Therefore, the framework now contains most of the reusable training machinery, but we do not currently provide a ready-to-run MaskedMimic RLFT experiment. The closest supported example today is the GPC PEFT SFT → RLFT pipeline (https://github.com/NVlabs/ProtoMotions/blob/main/docs/source/user_guide/gpc.rst).