Reproducing online SDPO results

[gautamsreekumar]

Hello,

I was trying to reproduce the results of online SDPO, specifically Figures 4 and 5 in ArXiv v1. The paper says the main model (Qwen3-4B) is trained on the TL;DR dataset with a Qwen3-8B as the user/judge model. The repository contains a script scripts/eval_online_sdpo.sh, which updates the model in an online fashion, essentially a batch size of 1. But in Appendix C, the batch size to obtain the results in Figures 4 and 5 is 16. There is a folder named online_simple_pg in the repo, but I am not sure if that folder was intended to be used as (1) it is not included in the repo directory structure at the end of README, (2) its files have some issues in their import lines, and (3) it uses a dummy reward function in the main file.

So, my questions are the following:

1. Can we obtain the results in Figures 4 and 5 using a batch size of 1, perhaps with another hyperparameter combination?
2. I tried a simple gradient accumulation to get an effective batch size of 16 using the provided eval_online_sdpo.sh by dividing the loss by 16 and calling the optimizer.step() method only every 16 steps. Is that a meaningful strategy without any changes to the learning rate? I noticed that online_simple_pg also used gradient accumulation.
3. How should the x-axis in Figures 4 and 5 be interpreted? Does "roll-outs/user interactions" refer to the number of samples or the number of optimization steps? The TL;DR dataset downloaded by the script in this repo has around 3900 training samples.