Clarification on GRPO training protocol (on-policy / per-step sample handling)

[NarcissusUse]

Hi MobileGym team,

First, thank you for this excellent work — the declarative task-definition framework, the deterministic outcome-based reward, and the browser-native simulator are a really clean and reproducible foundation for mobile-agent RL. We've been using MOBILEGYM-BENCH and the Sim-to-Real recipe as the basis for our own GRPO experiments and it's been a pleasure to build on.

We're trying to reproduce the Sim-to-Real result (Appendix G): Qwen3-VL-4B-Instruct, base SR 9.4% → 22.2% (+12.8 pt) with 10-step GRPO from base. We've matched the paper config as closely as we can:

Base model / base SR: Paper: Qwen3-VL-4B-Instruct / 9.4% | Ours: same model / 9.1% (our eval)
Learning rate: Paper: 1e-6, constant | Ours: 1e-6, constant
Group size k: Paper: 8 | Ours: 8
Rollout / train batch: Paper: 12 (12×8 = 96 envs) | Ours: 12 (total 96 envs)
Training set: Paper: 160 tasks | Ours: same 160 tasks
Temperature (train / eval): Paper: 0.7 / 0.1 | Ours: 0.7 / 0.1
Asymmetric clip range (DAPO): Paper: 0.2 / 0.28 | Ours: 0.2 / 0.28
KL coefficient: Paper: 0.01 | Ours: 0.01
Reward design: Paper: PR-shaped multiplicative reward (AnswerSheet / side-effect / false-complete / abort / overdue) | Ours: same formula
Training steps: Paper: 10 | Ours: 10

Despite matching these, our run goes 9.1% → 7.8% (−1.3 pt) instead of improving. So we suspect the gap is in the training protocol (how
rollouts are turned into gradient updates) rather than the hyperparameters, and we'd love to confirm a few points about how your GRPO
actually updates the policy.

Our setup is on the slime framework, where for multi-step agent episodes:

• each episode is expanded into per-step samples (one (obs, action) step = one training sample), with the episode-level reward/advantage
  broadcast onto every step;
• one rollout = 96 trajectories → flattened into all their step-samples → exactly one optimizer update over that flattened batch (we use a
  "dynamic global batch size" that forces num_steps_per_rollout = 1).

Questions:

1. Is your training strictly on-policy with exactly one optimizer update per rollout? With train batch size = 12 and PPO mini-batch size = 12
   being equal, it reads as a single mini-batch (i.e., one gradient step per rollout, importance ratio ≈ 1). Or do you run multiple PPO epochs
   / multiple mini-batch iterations over each rollout's data (so the policy drifts within a rollout and the clip actually engages)? This single
   number is the main thing we're unsure about.
2. For multi-step episodes, what is the training "sample" unit? Do you (a) flatten each trajectory into per-step samples and assign the
   episode reward/advantage to each step (what we do), (b) treat each whole trajectory as one sequence, or (c) something else? And is the GRPO
   group-normalization (over the k=8 rollouts of a task) done at the episode level before being applied to the steps?
3. Roughly how many total gradient updates happen over the 10 steps, and what does the per-rollout reward std / advantage signal
   look like (i.e., how often do the k=8 rollouts of a task share the same reward, giving zero advantage)? This would help us tell whether our
   gap is "too few updates" vs "no group-level signal from a 9% base."

Any pointers — or a release of the training config / script — would help us a lot. Thanks again for the great work!

[jiazhou-garland]

Hello, I've also recently been trying to reproduce this issue. When you reproduced the 9.4% evaluation rate in Qwen3-VL-4B-Instruct, what was the max_model_len setting? I've only found an SR of around 2.4% in my attempts. Could you please explain your evaluation parameters in detail?

[NarcissusUse]

Hello, I've also recently been trying to reproduce this issue. When you reproduced the 9.4% evaluation rate in Qwen3-VL-4B-Instruct, what was the max_model_len setting? I've only found an SR of around 2.4% in my attempts. Could you please explain your evaluation parameters in detail?
Hi, thanks for trying to reproduce this!

I ran into exactly the same thing — the base Qwen3-VL-4B-Instruct dropped to around 2.4% SR for me at first. In my case the culprit was max_tokens. When it's set too low, the model's responses get
truncated before it can emit a complete, parseable action, so a lot of otherwise-correct rollouts get scored as failures.

Setting max_tokens (max completion length) to 4096 fixed it for me and reproduced the ~9% SR. I'd suggest checking that value first — if yours is lower, that's very likely why you're seeing 2.4%.

Hope this helps, and let me know if it lines up on your side!

[jiazhou-garland]

Thank you for your reply. After setting max_tokens=4096, I still only achieved a SR of 2.8%.

I later discovered the problem was that the server lacked Chinese fonts, causing all Chinese characters in the rendered image to appear as squares.

After resolving this issue, my most recent SR was 8.98%.

Furthermore, I plan to reproduce GRPO in the future, and I will continue to follow your issue. You can also refer to the GRPO training details in the author's appendix as shown in the following pic.

[NarcissusUse]

Thank you for your reply. After setting max_tokens=4096, I still only achieved a SR of 2.8%.

I later discovered the problem was that the server lacked Chinese fonts, causing all Chinese characters in the rendered image to appear as squares.

After resolving this issue, my most recent SR was 8.98%.

Furthermore, I plan to reproduce GRPO in the future, and I will continue to follow your issue. You can also refer to the GRPO training details in the author's appendix as shown in the following pic.

Thank you, and glad to hear the font issue is resolved!

I've been training with exactly the parameters from the appendix, but there are two important details the paper doesn't specify:

1. How many optimizer updates are performed per rollout batch? This matters a lot because they report a +12pp gain within only 10 training steps — whether each batch of rollouts is used for a single gradient update or reused for multiple epochs/mini-batch updates (PPO-style) changes the interpretation of "10 steps" entirely.
2. How is each rollout batch processed before the update? And does the training-time input format need to match inference? Since inference takes only a single screenshot as input, do they split each trajectory into individual steps and broadcast the
   trajectory-level reward to every step? If so, one batch of rollouts could yield roughly 800–1000 (step, reward) samples per update. Or is the advantage computed and applied at the trajectory level instead (i.e., trajectory-level RL)?

If you figure out either of these during your GRPO reproduction, I'd really appreciate it if you could share your findings here. I'll
keep this issue updated with my progress as well.

[Purewhiter]

Hi, thanks for the detailed questions and reproduction notes.

The online RL training code has now been released in v0.1.0. Please take a look at mobilegym-rl/, especially:

• mobilegym-rl/README.md
• cookbooks/mobilegym/train_qwen3_vl_4b_verl.sh
• cookbooks/mobilegym/mobilegym_flow.py

These should answer the main protocol/configuration questions. If anything remains unclear, please feel free to follow up in this issue.