transformer_trainer_ddp.py: `DistributedSampler.set_epoch()` is commented out in both trainers, so the training data order is identical every epoch

[u7079256]

Summary

Both DDP training loops (mask transformer and residual transformer) build a torch.utils.data.distributed.DistributedSampler for the training loader, but the per-epoch train_sampler.set_epoch(epoch) call is commented out in both classes. Per the PyTorch documentation for DistributedSampler, set_epoch() must be called at the beginning of each epoch for shuffling to actually vary across epochs; otherwise the sampler uses the same seed every epoch and every rank iterates the dataset in exactly the same order for the entire run.

Affected code

• Sampler construction:

  ReMoMask/models/transformer/transformer_trainer_ddp.py

  Line 147 in 00499c3

  train_sampler = DistributedSampler(

• Commented-out call, mask-transformer trainer:

  ReMoMask/models/transformer/transformer_trainer_ddp.py

  Line 206 in 00499c3

  # train_sampler.set_epoch(epoch)

• Commented-out call, residual-transformer trainer:

  ReMoMask/models/transformer/transformer_trainer_ddp.py

  Line 463 in 00499c3

  # train_sampler.set_epoch(epoch)

                # train_sampler.set_epoch(epoch)

Expected vs actual behavior

• Expected: with a shuffling DistributedSampler, each epoch iterates the training data in a different permutation.
• Actual: without set_epoch, the permutation is identical in every epoch (the sampler's generator is seeded with the same value each time), for both train_mask_transformer_ddp.py and train_res_transformer_ddp.py. Training runs without any error or warning and the loss decreases normally, so nothing surfaces the problem.

Steps to reproduce

1. Add a one-line log of the first batch's sample indices at the top of each epoch in the training loop.
2. Run either training entry point for 2+ epochs with the shipped scripts.
3. Observe that the logged index sequence is identical across epochs; uncommenting train_sampler.set_epoch(epoch) makes it vary as expected.

Impact

The training schedules documented for this codebase run for hundreds to thousands of epochs. Over such a run, the model sees the training set in one fixed order the whole time. This silently changes the training procedure relative to standard practice and can affect final model quality; it also means the released checkpoints were trained under this fixed-order regime.