Cannot reproduce the published FID / R-Precision / MModality with the released checkpoints using the documented evaluation commands

[u7079256]

Summary

Evaluating the released HumanML3D checkpoints exactly as documented in the README, we measure metrics far from the published numbers (e.g. FID 0.123 ± 0.003 vs 0.026 camera-ready / 0.099 arXiv; MModality 1.45 vs 2.84). We are filing this as a reproducibility report, not a claim about the cause — we would appreciate the exact command and code path used to produce the published numbers.

Checkpoints and protocol (fully specified)

• Checkpoints (from the Hugging Face release; file sizes identical between https://huggingface.co/lycnight/ReMoMask and https://huggingface.co/AIGeeksGroup/ReMoMask):
  • logs/humanml3d/pretrain_mtrans/model/net_best_fid.tar (mask transformer, epoch 616 per its own metadata)
  • logs/humanml3d/pretrain_rtrans/model/net_best_fid.tar (residual transformer)
  • logs/humanml3d/pretrain_vq/model/net_best_fid.tar (RVQ-VAE), evaluator checkpoints under checkpoints/humanml3d/ as shipped
• Retrieval database: rebuilt with build_rag_database.py + the released Part_TMR/checkpoints/exp_for_mtrans checkpoint over the HumanML3D training split (necessary because the shipped database/ contains only 32 entries — reported separately).
• Evaluation command: the README's documented invocation (

  ReMoMask/README.md

  Lines 233 to 248 in 00499c3

  	### 3. Evaluate the 2D Residual Transformer
  	HumanML3D:
  	```bash
  	python eval_res.py \
  	--gpu_id 0 \
  	--dataset_name humanml3d \
  	--mtrans_name pretrain_mtrans \
  	--rtrans_name pretrain_rtrans \
  	--cond_scale 4 \
  	--time_steps 10 \
  	--ext eval \
  	--which_ckpt net_best_fid.tar \
  	--which_epoch fid \
  	--traverse_res
  	# change pretrain_mtrans and pretrain_rtrans to your mtrans and rtrans
  	```

  ):

python eval_res.py \
    --gpu_id 0 \
    --dataset_name humanml3d \
    --mtrans_name pretrain_mtrans \
    --rtrans_name pretrain_rtrans \
    --cond_scale 4 \
    --time_steps 10 \
    --ext eval \
    --which_ckpt net_best_fid.tar \
    --which_epoch fid

plus a fixed random seed of 10107. eval_res.py internally runs 20 evaluation repeats; we report mean ± 1.96×SEM over those 20 repeats. Mask-transformer-only numbers below use the README's eval_mask.py command with the same cond_scale/time_steps/seed and 20 repeats.

• Environment: single NVIDIA L4 GPU, PyTorch 2.x, HumanML3D test split, metrics computed by the repository's own evaluation code, unmodified.

Expected vs actual results

metric	measured (released ckpts, 20 repeats)	published (camera-ready)	published (arXiv)
FID (full pipeline, eval_res.py)	0.123 ± 0.003	0.026	0.099
FID (mask transformer only, eval_mask.py)	0.143 ± 0.005	—	—
Top-1 R-Precision	0.485	0.566	0.531
Matching score	3.090 ± 0.008	2.867	2.865
MModality	1.446 ± 0.045	2.835	2.823

The MModality discrepancy (1.45 vs 2.84) is particularly hard to attribute to environment differences on a fixed checkpoint.

Observation (not a claim)

The codebase has three code paths that print an FID:

1. eval_res.py — full text-to-motion generation + residual refinement (measured above);
2. eval_mask.py — text-to-motion generation without refinement (measured above);
3. evaluation_res_transformer in utils/eval_t2m_ddp.py, used during residual-transformer training — it starts from ground-truth motion tokens (

   ReMoMask/utils/eval_t2m_ddp.py

   Line 394 in 00499c3

   def evaluation_res_transformer(out_dir, val_loader, trans_aux, trans_ts, vq_model, writer, ep,

   and

   ReMoMask/utils/eval_t2m_ddp.py

   Line 450 in 00499c3

   code_indices_aux, _, code_indices_ts, _ = vq_model.encode(motion_gt, joints_gt)

   , vq_model.encode(motion_gt, joints_gt)), i.e. it measures refinement on top of ground truth rather than text-to-motion generation.

We note only that the released residual-transformer checkpoint's internally recorded best FID lies in the 0.022-0.030 range, numerically close to the published 0.026 — and ask for clarification of which path produced the published numbers.

Supporting evidence: no shipped trace of how these checkpoints were evaluated

Via the Hugging Face Hub API (GET /api/models/lycnight/ReMoMask?blobs=true), the log files shipped alongside exactly these checkpoints are empty or absent:

file	size (bytes)
logs/humanml3d/pretrain_mtrans/train.log	0
logs/humanml3d/pretrain_mtrans/eval/eval.log	0
logs/humanml3d/pretrain_rtrans/eval/eval.log	0
logs/humanml3d/pretrain_rtrans/train.log	(not in the repo)
logs/humanml3d/pretrain_vq/eval/eval.log	0
Part_TMR/checkpoints/exp_for_mtrans/HumanML3D/train.log	0

(For contrast, Part_TMR/checkpoints/exp_pretrain/HumanML3D/train.log in the same repo has 35,289 bytes of real content.) So the discrepancy cannot be resolved from the released artifacts alone.

[u7079256]

What would help

The exact command line (script + all flags, and which of the three code paths above) and checkpoint files used to produce the camera-ready FID 0.026 and the arXiv FID 0.099, so we can attempt an exact reproduction.