Reproducibility issues

[MaxHaufe]

Hi,
thank you for your work and open-sourcing the code.
I have tried to reproduce the paper using the code provided but ran into several issues as described below. Could you please clarify if I am missing something?

Missing test set for T2 Slip GelSight

These are the configurations for train and test according to the config for T2_gelsight.

     list_datasets: [
          sphere/batch_1,
          sphere/batch_3,
          sphere/batch_4,
          sphere/batch_5,
          sphere/batch_6,
      ]
      list_datasets_test: [
          sphere/batch_test,
      ]

The dataset provided (link) does not include batch_test. Instead I renamed batch_2 to batch_test. This might not matter much as T2 for GelSight was the only task where my results did not differ by much from those reported in the paper.

Missing preprocessing scripts for T5 textile

I was not able to reproduce the results for T5 for the reason described in #12

Inconsistent results within the paper

This is for for E2E and T2_slip_digit

Main text	Appendix

The result for the E2E model in the main text (Figure 4, page 7) differs from the result reported in the appendix (Figure 17b, page 30). Which one is the correct one?

Inconsistent results reproduced by myself

Now I will report the results reproduced by myself using the provided code and datasets. I compare these to the results reported in the appendix in Figure 17.

T1_force_digit

Sparsh appendix	My result

For 0.01 data budget, my E2E and MAE results are both lower than reported in the paper.

T1_force_gelsight

Sparsh appendix	My result

My results are vastly different.

T2_slip_digit

Sparsh appendix	My result

My E2E result looks much different.

T2_slip_gelsight

Sparsh appendix	My result

This one looks good.

T3_pose_digit

Sparsh appendix	My result

This one is very off. My results are 20% worse.

T4_grasp_gelsight

Sparsh appendix	My result

This one looks off as well, mostly the E2E one.

Notes

• I have downloaded the data from huggingface and the links provided in the readme. I used the provided scripts to pre-process the data.
• I submit the jobs via the submit_task.sh script.
• My HPC cluster uses NVIDIA H100s
• I do not use xFormers. ($XFORMERS_DISABLED=1)
• I might have a different pytorch version; I had to change torch.load(checkpoint_task) to torch.load(checkpoint_task, weights_only=False, map_location="cpu") in tactile_sll/downstream_task/sl_module.py
• My pytorch version: 2.8.0+cu128. No idea why it isn't 2.0.*, I used the provided environment.yaml to install.
• Training downstream tasks appears to be nondeterministic, even though everything is seeded. I believe (although I am not 100% sure) that this is due to torch.backends.cudnn.benchmark = True. Starting multiple training runs with the same code and configuration yields slightly different training losses.

Please let me know if you have an idea on where the issue might be coming from. Maybe you have observed the nondeterminism as well, maybe this section points towards this.

Thanks,
Max