Different reproduced results

[purentap]

Hi,

I collected the simulation data by using your repositories as it is. When I reproduce the results using your bash script, I get the following:

for pi0:
Indep:
seen : \mstd{81.52}{3.26}
unseen: \mstd{80.23}{10.78}

LSTM:
seen: \mstd{85.19}{1.68}
unseen: \mstd{81.09}{10.26}

for open-pi0:
Indep:
seen: \mstd{95.87}{1.07}
unseen: \mstd{82.14}{7.05}

LSTM:
seen: \mstd{92.88}{0.91}
unseen: \mstd{74.26}{7.41}

For openvla experiments, the unseen score drops around 5.

What could be the reason for this drastic changes? Am I missing something? I though for pi0, it could be due to newer version of the fine-tuned models but after seeing the change in simplerenv version as well, im not so sure. Is the seed in the data collection still differs the collected data?

[georgegu1997]

Hi, sorry for the late reply!

Yes, we also notice that the results may fluctuate due to random seeding or package version discrepancies. This can also be noted from the high variance in the results (5-10 on unseen splits across the board).

It may also be due to the discrepancies and randomness in data generation. For reference, here are the links to the data we used.

OpenVLA + LIBERO: See post below

open-pi-zero on SimplerEnv: See post below.

pi0 on LIBERO: See post below

pi0-faster on LIBERO: See post below

Please have a try on this data.

If you still notice a difference and want an exact reproduction, we can also help! Please then paste the full command you ran to get your results. We can try to reproduce it and see whether there is anything wrong.

[gyyzuiaizyl-code]

I am very interested in your recent work and was hoping to utilize the resources you shared. However, it seems the SharePoint links have expired and are no longer accessible.

Specifically, the following links are invalid:

OpenVLA + LIBERO: [Delete by @georgegu1997 to avoid confusion]

open-pi-zero on SimplerEnv: [Delete by @georgegu1997 to avoid confusion]

pi0 on LIBERO: [Delete by @georgegu1997 to avoid confusion]

pi0-faster on LIBERO: [Delete by @georgegu1997 to avoid confusion]

Could you please kindly update these links or let me know if there is an alternative platform to download them?

Thank you so much for your time and for your contribution to the community!

[georgegu1997]

Hi, I refreshed the links below. Please have a try on these links:

OpenVLA + LIBERO: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQCdNzlCN01lS7pHbtxrF4DKAfpjrNmKfotZemzg46AsnWA?e=U6UW3E

open-pi-zero on SimplerEnv: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQCHKWZxCXsTTrFHxIyHYWe6Ae-UnSkp-OjE0bU0ZbARmQQ?e=mByZfe

pi0 on LIBERO: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQBPGVs_AKtPRIKhgegHgeI1AYZTbap3KaIrpAhCHRfbcaY?e=kbantD

pi0-fast on LIBERO: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQCXvWCXfJZARKS-tGZ3GBzxAWK5fV86tDG37RBs6zq96Q8?e=3K94wm

Sorry about the expiration. My university OneDrive forces public sharing to have an expiration date. I will try to find a more permanent way to share the dataset.

[gyyzuiaizyl-code]

Dear Qiao， I hope you are doing well. First of all, thank you very much for providing the official rollout data. It has been very helpful for reproducing the paper results and understanding the evaluation pipeline. I am trying to reproduce the π0-FAST + LIBERO-10 SAFE-MLP results reported in your paper, and I would like to ask for your help in checking whether I have misunderstood the reproduction protocol or the hyperparameter table. I used the official pi0fast-libero.zip rollout file that you provided. After extracting it, I verified the rollout structure: Item Value env_records 500 policy_records 35,867 Total success rate 298 / 500 = 59.6% Policy step IDs continuous from step_0 to step_35866 sum(model_infer_times) 35,867 Prompt / task-description alignment all 35,867 policy records matched I also checked the task-wise success rates: Task Success / 50 Failure / 50 Success Rate 0 42 8 84.0% 1 47 3 94.0% 2 18 32 36.0% 3 19 31 38.0% 4 45 5 90.0% 5 24 26 48.0% 6 39 11 78.0% 7 49 1 98.0% 8 0 50 0.0% 9 15 35 30.0% The overall success rate is close to the paper, but the task-level distribution is quite imbalanced, especially with task 8 having 0/50 successful episodes. I then reproduced SAFE-MLP using a clean clone of the official SAFE repository: Repo: https://github.com/vla-safe/SAFE.git Commit: b6036ab Rollout: official pi0fast-libero_10 Dataset: pizero_fast Feature: pre_logits Token aggregation: mean Model: indep / SAFE-MLP Seeds: 0, 1, 2 According to my reading of Table 9, the best π0-FAST + LIBERO SAFE-MLP hyperparameters seem to be: feat = pre_logits aggtoken = mean lr = 1e-4 lambda_reg = 1e-2 However, when I fixed these hyperparameters and ran three seeds, I obtained a very different result: Setting Seed Seen ROC-AUC Unseen ROC-AUC lr=1e-4, lambda_reg=1e-2 0 85.64 39.85 lr=1e-4, lambda_reg=1e-2 1 56.68 71.40 lr=1e-4, lambda_reg=1e-2 2 32.55 93.15 Mean ± std - 58.29 ± 21.70 68.13 ± 21.88 This was surprising because it is far from the reported π0-FAST SAFE-MLP result: Source Seen ROC-AUC Unseen ROC-AUC Paper, π0-FAST SAFE-MLP 90.06 ± 2.82 80.44 ± 5.72 I then tried another fixed hyperparameter setting, still using the official rollout and the clean SAFE repository: feat = pre_logits aggtoken = mean lr = 3e-5 lambda_reg = 1e-3 This gave a result much closer to the paper: Setting Seed Seen ROC-AUC Unseen ROC-AUC lr=3e-5, lambda_reg=1e-3 0 93.86 74.68 lr=3e-5, lambda_reg=1e-3 1 88.15 72.85 lr=3e-5, lambda_reg=1e-3 2 84.88 92.84 Mean ± std - 88.96 ± 3.71 80.12 ± 9.02 This is close to the reported Table 1 result, especially for unseen ROC-AUC. However, I noticed that 3e-5 seems to appear in the LSTM sweep range in the script, while the MLP sweep script I checked uses: model.lr = 1e-5, 1e-4, 3e-4, 1e-3 model.lambda_reg = 1e-3, 1e-2, 1e-1 Therefore, I am not sure whether I am misunderstanding Table 9, using the wrong selection protocol, or missing some detail in the evaluation procedure. Could you please help me clarify the following questions? 1. For the π0-FAST + LIBERO SAFE-MLP result in Table 1, are the hyperparameters fixed to the bold values in Table 9 for all three seeds, or are they selected separately for each seen/unseen split using D_eval-seen ? 2. For SAFE-MLP on π0-FAST, should the learning-rate search range include 3e-5, or should it only use 1e-5, 1e-4, 3e-4, 1e-3 as in the current script? 3. Is the reported Table 1 result computed from the final epoch, the best checkpoint according to D_eval-seen, or the final metrics of the selected hyperparameter run? 4. Is the official pi0fast-libero.zip rollout the same data used for the Table 1 π0-FAST LIBERO result? 5. Given that task 8 is 0/50 successful in this rollout, is the evaluation intended to use pooled ROC-AUC over all episodes in each split, or task-wise averaged ROC-AUC? I would really appreciate any guidance. My current best reproduction is close to the paper numerically, but it uses lr=3e-5, lambda_reg=1e-3, while my reading of Table 9 suggested lr=1e-4, lambda_reg=1e-2. Thank you very much for your time and help. Best regards Qiao Gu ***@***.***> 于2026年5月20日周三 04:01写道：

…

*georgegu1997* left a comment (vla-safe/SAFE#5) <#5 (comment)> Hi, I refreshed the links below. Please have a try on these links: OpenVLA + LIBERO: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQCdNzlCN01lS7pHbtxrF4DKAfpjrNmKfotZemzg46AsnWA?e=U6UW3E open-pi-zero on SimplerEnv: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQCHKWZxCXsTTrFHxIyHYWe6Ae-UnSkp-OjE0bU0ZbARmQQ?e=mByZfe pi0 on LIBERO: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQBPGVs_AKtPRIKhgegHgeI1AYZTbap3KaIrpAhCHRfbcaY?e=kbantD pi0-fast on LIBERO: https://utoronto-my.sharepoint.com/:u:/g/personal/q_gu_mail_utoronto_ca/IQCXvWCXfJZARKS-tGZ3GBzxAWK5fV86tDG37RBs6zq96Q8?e=3K94wm Sorry about the expiration. My university OneDrive forces public sharing to have an expiration date. I will try to find a more permanent way to share the dataset. — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BXCLXMFQ4C4QGEBHCIEYDVD43S4TNAVCNFSM6AAAAACSX7X77KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIOJRGU2TKOBRGU> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.Message ID: ***@***.***>

[uuu231]

Hi, thanks for your great work.
I'm trying to reproduce the result from Figure 6(a) using data from https://drive.google.com/file/d/13z_cdwnaJota2iHkZbhYgVALujZwtM3b/view?usp=sharing and training script submit_pi0fast_droid.bash.
I set model.n_layers = 2, hidden_dim = 256, model.lr = 3e-4, and lambda_reg = 1e-3 for the safe-mlp model, which is the optimal hyperparameters setting from Table 11.
After training for 1000 epochs with 5 random seeds, I got:
for dataset.token_idx_rel=mean
falert_early_roc_auc/model_val_seen: 75.58
falert_early_roc_auc/model_val_unseen: 58.27

I also tried other hyperparameters settings but couldn't reach 86.76 (val_seen) / 64.16 (unseen). I notice not all data subsets are being used — should I adjust other hyperparameters or the dataset config?

Hi, sorry for the late reply!

Yes, when we collected the rollouts in the real world, we set a maximum length for each task, and each rollout will terminate when that time is reached. However, there are also some rollouts that we terminated early due to various reasons (not progressing at all, failing in a dangerous way, or something else occurring). Therefore, not all rollouts in each task have the same length.

For SAFE, we only use the rollouts that have the full length (max length in each task). Please refer to this snippet: https://github.com/vla-safe/SAFE/blob/main/failure_prob/data/pizero_fast_droid.py#L416

Also, we didn't use all the rollouts in training and evaluation but sampled a subset. Please also see the above dataloading script for details.