Evaluating MolmoBot-RBY1Multitask on RB-Y1 pick and pick-and-place benchmarks

[jinyoonok2]

Dear MolmoBot team,

Thank you for your work on these frameworks.

I am trying to reproduce the RB-Y1 multitask simulation evaluation results reported on the MolmoBot technical website:

MolmoBot technical website

The website reports the following RB-Y1 simulation success rates for MolmoBot Multitask:

• Pick: 44.8%
• Pick & Place: 22.5%
• Open: 25.2%
• Door Open: 70.2%

I am using the Hugging Face model:

MolmoBot-RBY1Multitask Hugging Face model

and the evaluation/configuration code from the public MolmoBot and MolmoSpaces repositories.

However, I have not been able to reproduce the reported RB-Y1 Pick and Pick-and-Place performance. In my runs, Pick and Pick-and-Place success rates are much lower than the reported values, close to 0, while Opening and Door Opening show nonzero/decent success rates.

One issue I encountered is that the documented/evaluation entrypoint does not seem to work cleanly with the current RB-Y1 multitask benchmark setup. When using the MolmoBot launch_scripts/run_eval.py path with the RB-Y1 multitask eval config, I encountered compatibility errors between MolmoBot and molmospaces, including arguments passed to run_evaluation() that are not accepted by the current molmospaces API, plus some RB-Y1 policy/config mismatches.

Because of this, I created a small separate evaluation wrapper that directly calls the RB-Y1 multitask policy/eval configuration so I could run the benchmark episodes. My intent was not to change the model behavior, but only to work around the launcher/API mismatch and evaluate the released MolmoBot-RBY1Multitask checkpoint on the RB-Y1 benchmark.

I also attached four example rollout videos from two failed episodes:

• Pick example: benchmark index 10

  • Full task instruction: pick up the knocker.
  • Target object: dark metal knocker with figure
  • Attached views: camera_follower and wrist_camera_r
  • Episode result: 0/1, failed

• Pick-and-Place example: benchmark index 21

  • Full task instruction: pick up the clear cup and place it into/on the round ceramic bowl with floral patterns
  • Pickup target: tall transparent ridged cylindrical cup
  • Place target: round ceramic bowl with floral patterns
  • Attached views: camera_follower and wrist_camera_r
  • Episode result: 0/1, failed
  • In the rollout video, the object appears to be gripped/lifted at some point, but the episode is still marked as failed by the benchmark success condition.

Pick idx 10, camera_follower:

https://github.com/user-attachments/assets/90ff0bc8-fcf3-4fdc-9545-aa6f63c38de8

Pick idx 10, wrist_camera_r:

episode_00000000_wrist_camera_r_batch_1_of_1.mp4

Pick-and-Place idx 21, camera_follower:

episode_00000000_camera_follower_batch_1_of_1.mp4

Pick-and-Place idx 21, wrist_camera_r:

episode_00000000_wrist_camera_r_batch_1_of_1.mp4

Evaluation Summary

All runs used MolmoBot-RBY1Multitask on RB-Y1 benchmarks from molmospaces-bench-v2/20260415.

Baseline Reproduction Results

Run	Tasks	Episodes	Overall	Pick	Pick & Place	Opening	Door Opening	Successful episodes
30-index baseline	pick, pnp, opening, door_opening	120	25/120 (20.8%)	0/30 (0.0%)	1/30 (3.3%)	9/30 (30.0%)	15/30 (50.0%)	pnp:225; opening:0,1,2,8,9,225,325,725,775; door_opening:6,7,8,9,25,75,125,175,325,375,475,625,725,775,825
Pick/PnP expansion	pick, pnp	140	2/140 (1.4%)	0/70 (0.0%)	2/70 (2.9%)	-	-	pnp:58,304

Additional Horizon Test

All tests used the same 10 shared Pick/PnP indices: 10, 21, 33, 56, 58, 74, 99, 101, 123, 137.

Experiment	Changed parameter(s)	Overall	Pick	Pick & Place	Successful episodes
horizon 450	task_horizon_steps=450	0/20 (0.0%)	0/10 (0.0%)	0/10 (0.0%)	none

Episode Index Sets

Run group	Tasks	Episode indices
30-index baseline	pick, pnp, opening, door_opening	0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 25, 75, 125, 175, 225, 275, 325, 375, 425, 475, 525, 575, 625, 675, 725, 775, 825, 875, 925, 975
Pick/PnP expansion	pick	10, 21, 33, 56, 58, 74, 99, 101, 123, 137, 155, 158, 171, 199, 206, 224, 230, 255, 268, 279, 291, 304, 318, 334, 354, 361, 384, 388, 413, 420, 438, 447, 470, 472, 488, 504, 521, 533, 546, 566, 578, 589, 600, 619, 640, 649, 661, 682, 687, 713, 715, 729, 748, 758, 782, 795, 804, 821, 829, 845, 866, 872, 892, 900, 921, 942, 947, 961, 974, 986
Pick/PnP expansion	pnp	10, 21, 33, 56, 58, 74, 99, 101, 123, 137, 155, 158, 171, 199, 206, 224, 230, 255, 268, 279, 291, 304, 318, 334, 354, 361, 384, 388, 413, 420, 438, 447, 470, 472, 488, 504, 521, 533, 546, 566, 578, 589, 600, 619, 640, 649, 661, 682, 687, 713, 715, 729, 748, 758, 782, 795, 804, 821, 829, 845, 866, 872, 892, 900, 921, 942, 947, 961, 974, 986
Horizon test	pick, pnp	10, 21, 33, 56, 58, 74, 99, 101, 123, 137

Opening and Door Opening show nonzero success rates, but I still could not reproduce the reported Pick/PnP performance. In the completed Pick/PnP expansion, Pick remained 0/70 and Pick-and-Place was 2/70.

Could you clarify the exact evaluation method used to produce the RB-Y1 simulation results on the website?

In particular:

1. What exact command or script should be used to evaluate MolmoBot-RBY1Multitask on RB-Y1 Pick and Pick-and-Place?

2. Which benchmark version should be used?
   I found that molmospaces-bench-v2/20260415 includes RB-Y1 benchmarks. Is this the benchmark version used for the reported RB-Y1 results, or was another benchmark/version used?

3. Are there required evaluation parameters for RB-Y1 multitask evaluation?
   For example:

   • task horizon
   • renderer settings
   • terminate-on-success behavior
   • required eval config class
   • any special policy config overrides

4. Are the released Hugging Face checkpoint and current public code expected to reproduce the reported RB-Y1 Pick and Pick-and-Place success rates directly?

I may be using the wrong benchmark or missing an evaluation setting, so I would appreciate clarification on the intended reproduction path for the RB-Y1 multitask results.

Thank you!

[jinyoonok2]

Additional check: replaying released MolmoBot-data trajectories

@abhaybd

Dear Molmobot team,

I also tried a separate sanity check using the released allenai/molmobot-data trajectories.

My goal was to test whether recorded expert/data-generation trajectories could be replayed successfully in the current public MolmoSpaces simulation stack. I used validation trajectories from: molmobot-data

• RBY1PickDataGenConfig
• RBY1PickAndPlaceDataGenConfig
• RBY1OpenDataGenConfig

I also generated compatibility replay benchmark JSONs from the downloaded allenai/molmobot-data metadata so that the current MolmoSpaces code could initialize the corresponding scenes/tasks. The original recorded actions in the source .h5 files were not modified; during replay, the policy read those recorded actions directly and applied them in MolmoSpaces.

Using this replay setup, the outputs were not consistently successful in the current public codebase. This affected Pick, Pick-and-Place, and Open replay episodes: some recorded trajectories did not satisfy the expected success checks, even though they appear to be expert/data-generation demonstrations.

open drawer episode example:

open cabinet episode example:

I do not interpret this as evidence that the original trajectories were unsuccessful. It may instead indicate a mismatch between the environment used to generate the released trajectories and the current public evaluation stack, such as:

• simulator / MolmoSpaces version differences
• asset or collision geometry differences
• controller or command-mode differences
• success-condition differences
• replay horizon or timing differences
• benchmark/task config differences

This makes me think the reproduction issue may not be only model-side. It may also involve the exact simulator, benchmark, controller, or task configuration used for the reported RB-Y1 results.

Could you clarify whether the released allenai/molmobot-data RBY1 trajectories are expected to replay successfully in the current public MolmoSpaces codebase? If so, is there an official replay/evaluation script or required config for doing this?

[sjauhri]

Hi @jinyoonok2
Apologies for the late response.
For the molmospaces version, please use the version pinned in this repo:

MolmoBot/MolmoBot/pyproject.toml

Line 79 in 33c0ca7

"molmo_spaces[mujoco] @ git+https://github.com/allenai/molmospaces.git@cd23becebcf72dd93a4aa5872a60802d5eff03ef#egg=molmo_spaces[mujoco]",

The current molmobot code supports that version.

This is also true for the molmobot data replays.

[jinyoonok2]

Dear @sjauhri ,

I really appreciate your reply and for pointing me to the MolmoSpaces version pinned in MolmoBot/MolmoBot/pyproject.toml. I updated my setup to use the pinned MolmoSpaces commit:

cd23becebcf72dd93a4aa5872a60802d5eff03ef

I then reran the RB-Y1 simulation evaluation using the released multitask checkpoint:

MolmoBot-RBY1Multitask

For this rerun, I evaluated 100 benchmark episodes per task family using episode indices:

0, 10, 20, ..., 990

I ran the evaluations through the MolmoBot simulation eval path, calling MolmoBot/MolmoBot/launch_scripts/run_eval.py one episode at a time with the following key arguments:

python launch_scripts/run_eval.py \
  --checkpoint_path /u/xna8aw/workspace/live-robotics-lab/hf_models/MolmoBot-RBY1Multitask \
  --benchmark_path <benchmark_dir> \
  --eval_config_cls <eval_config_cls> \
  --output_dir <output_dir> \
  --num_workers 1 \
  --idx <episode_index> \
  --task_horizon 400 \
  --terminate_upon_success

The task-specific benchmark paths and eval config classes were:

Pick:
  benchmark: procthor-objaverse/rby1_bennchmarks/pick_benchmark
  eval_config_cls: olmo.eval.configure_molmo_spaces:MolmoBotRBY1PickPnPEvalConfig

Pick-and-Place:
  benchmark: procthor-objaverse/rby1_bennchmarks/pnp_benchmark
  eval_config_cls: olmo.eval.configure_molmo_spaces:MolmoBotRBY1PickPnPEvalConfig

Opening:
  benchmark: ithor/rby1_bennchmarks/door_opening_benchmark
  eval_config_cls: olmo.eval.configure_molmo_spaces:MolmoBotRBY1DoorPlusOpenEvalConfig

Door Opening:
  benchmark: procthor-10k/rby1_bennchmarks/door_opening_benchmark
  eval_config_cls: olmo.eval.configure_molmo_spaces:MolmoBotRBY1DoorPlusOpenEvalConfig

The 100-episode results were:

Task	Episodes	Success	Failure	Unknown
Pick	100	0	99	1
Pick-and-Place	100	23	77	0
Opening	100	77	23	0
Door Opening	100	77	23	0

Using the pinned MolmoSpaces version improved the Opening and Door Opening behavior significantly in my setup. However, Pick and Pick-and-Place still appear to struggle compared with the success rates reported on the MolmoBot technical website. Pick is the most concerning: it failed almost every evaluated episode, while Pick-and-Place had partial success.

I wanted to ask if these Pick/Pick-and-Place numbers are expected for the released MolmoBot-RBY1Multitask checkpoint with the pinned MolmoSpaces version, or if there may still be some setup mismatch on my side.

In particular, could you clarify:

1. Whether the benchmark paths above are the correct RB-Y1 benchmarks for reproducing the website results.

2. Whether there is a corresponding MolmoBot commit/version that should be used together with the pinned MolmoSpaces commit cd23becebcf72dd93a4aa5872a60802d5eff03ef.

3. Whether Pick and Pick-and-Place require any additional eval parameters or policy config settings beyond MolmoBotRBY1PickPnPEvalConfig.

4. Whether the released Hugging Face checkpoint is expected to reproduce the reported Pick and Pick-and-Place success rates directly with the public code and pinned MolmoSpaces commit.

I may still be missing a benchmark/resource version, MolmoBot code version, or evaluation setting, so I would appreciate any clarification on the intended reproduction path for the RB-Y1 multitask Pick and Pick-and-Place results.

Thank you again!

[sjauhri]

Hi @jinyoonok2
That's good to hear. The opening and pnp results seem ok now. For pick I will double check what the issue could be since the success criteria there has been changed a few times. Can you render the videos of your pick runs to see if they are atleast looking reasonable?

[sjauhri]

Could you also try running the pick eval without the last two settings? (--task_horizon 400 and --terminate_upon_success)?

[sjauhri]

Also, FYI, the opening task is actually the opening of other articulated objects besides doors. For that, try:
benchmark: ithor/rby1_benchmarks/opening_benchmark
eval_config_cls: olmo.eval.configure_molmo_spaces:MolmoBotRBY1DoorPlusOpenEvalConfig

[jinyoonok2]

Hi @sjauhri,

Thank you so much for the suggestions. I tried the additional evaluation setting change you mentioned and also inspected representative Pick and Pick-and-Place rollout videos.

Follow-up Evaluation Change

You suggested trying Pick eval without the last two settings:

--task_horizon 400
--terminate_upon_success

I first tried removing both. However, in my current setup, removing --task_horizon 400 fails before rollout because the benchmark JSON episodes do not contain task_horizon_sec:

ValueError: No explicit task_horizon override provided and 2000/2000 benchmark episodes are missing 'task_horizon_sec'. Either add task_horizon_sec to the benchmark JSON or pass --task_horizon_steps / --task_horizon_sec explicitly.

So I could not run without an explicit task horizon unless I patch the benchmark metadata or the MolmoSpaces horizon fallback behavior.

As a runnable follow-up, I kept an explicit horizon but changed the setting to:

--task_horizon 600

and removed:

--terminate_upon_success

I then reran 100 episodes per task family using the same episode indices as before:

0, 10, 20, ..., 990

The command structure was:

python launch_scripts/run_eval.py \
  --checkpoint_path /u/xna8aw/workspace/live-robotics-lab/hf_models/MolmoBot-RBY1Multitask \
  --benchmark_path <benchmark_dir> \
  --eval_config_cls <eval_config_cls> \
  --output_dir <output_dir> \
  --num_workers 1 \
  --idx <episode_index> \
  --task_horizon 600

No --terminate_upon_success flag was passed.

Results With task_horizon=600 and No terminate_upon_success

Task	Episodes	Success	Failure	Unknown	Success Rate
Pick	100	0	100	0	0.0%
Pick-and-Place	100	13	87	0	13.0%
Opening	100	56	44	0	56.0%
Door Opening	100	56	44	0	56.0%

Compared with the previous task_horizon=400 + --terminate_upon_success run:

Task	Previous Setting	New Setting
Pick	0/100	0/100
Pick-and-Place	23/100	13/100
Opening	77/100	56/100
Door Opening	77/100	56/100

So in my setup, this change did not improve Pick or Pick-and-Place. Pick still failed all evaluated episodes, and Pick-and-Place decreased from 23% to 13%.

Benchmark/Data Source

For the benchmark resources, I used the MolmoSpaces benchmark resources downloaded through the public MolmoSpaces resource system

The resource paths I currently have available are:

procthor-objaverse/rby1_bennchmarks/pick_benchmark
procthor-objaverse/rby1_bennchmarks/pnp_benchmark
ithor/rby1_bennchmarks/door_opening_benchmark
procthor-10k/rby1_bennchmarks/door_opening_benchmark

These came from the MolmoSpaces benchmark resources in the public Hugging Face dataset:
allenai/molmospaces

Specifically, I used:

mujoco/benchmarks/molmospaces-bench-v2/20260325_1

However, I could not find the specific benchmark path you mentioned:

ithor/rby1_benchmarks/opening_benchmark

In my downloaded pinned benchmark cache, the available iTHOR RBY1 path is instead:

ithor/rby1_bennchmarks/door_opening_benchmark

Could you verify which specific dataset/resource/version contains:

ithor/rby1_benchmarks/opening_benchmark

and whether it should come from MolmoSpaces resources, allenai/molmobot-data, or another benchmark archive?

Representative Videos

Here are the videos from the pick and pnp task runs, I found that most of them are trying to perform the task instead of completely misunderstanding them.

It seems that the arm orientation was not successful enough to complete the task.

I selected three failed Pick examples and two failed Pick-and-Place examples. For each one, I will attach both the scene/camera-follower view and the wrist-camera view.

Pick Examples

Pick example 1: index 0

Task instruction:

Pick up the blue puzzle piece with rainbow

Target object:

blue puzzle piece with rainbow

Episode result:

failed

This one was ambiguous because the robot seems to succeed the task, the evaluation still said it is failure.

Scene camera / camera_follower video:

pick_idx_0_failed_blue_puzzle_piece_with_rainbow_camera_follower.mp4

Wrist camera / wrist_camera_r video:

pick_idx_0_failed_blue_puzzle_piece_with_rainbow_wrist_camera_r.mp4

Pick example 2: index 10

Task instruction:

Pick up the dark metal knocker with figure

Target object:

dark metal knocker with figure

Episode result:

failed

Scene camera / camera_follower video:

pick_idx_10_failed_dark_metal_knocker_with_figure_camera_follower.mp4

Wrist camera / wrist_camera_r video:

pick_idx_10_failed_dark_metal_knocker_with_figure_wrist_camera_r.mp4

Pick example 3: index 50

Task instruction:

Pick up the blue radio with antenna

Target object:

blue radio with antenna

Episode result:

failed

Scene camera / camera_follower video:

pick_idx_50_failed_blue_radio_with_antenna_camera_follower.mp4

Wrist camera / wrist_camera_r video:

pick_idx_50_failed_blue_radio_with_antenna_wrist_camera_r.mp4

Pick-and-Place Examples

Pick-and-Place example 1: index 10

Task instruction:

Pick up the black cat silhouette keychain with ring and place it in or on the light ceramic shallow round bowl

Pickup target:

black cat silhouette keychain with ring

Place target:

light ceramic shallow round bowl

Episode result:

failed

Scene camera / camera_follower video:

pnp_idx_10_failed_objakeychain_8531d69cbac248dd9e90150e14af0af7_1_0_6_camera_follower.mp4

Wrist camera / wrist_camera_r video:

pnp_idx_10_failed_objakeychain_8531d69cbac248dd9e90150e14af0af7_1_0_6_wrist_camera_r.mp4

Pick-and-Place example 2: index 50

Task instruction:

Pick up the red apple and place it in or on the light ceramic shallow round bowl

Pickup target:

red apple

Place target:

light ceramic shallow round bowl

Episode result:

failed

Scene camera / camera_follower video:

pnp_idx_50_failed_apple_1184bcea45a878476443040fbba5b76d_1_0_6_camera_follower.mp4

Wrist camera / wrist_camera_r video:

pnp_idx_50_failed_apple_1184bcea45a878476443040fbba5b76d_1_0_6_wrist_camera_r.mp4

Thanks again for your help.

[jinyoonok2]

Hi @abhaybd,

Sorry to bother you again. I wanted to follow up because I still have not been able to find the benchmark path you mentioned.

I checked the available public allenai/molmospaces benchmark versions, including:

molmospaces-bench-v1/20260210
molmospaces-bench-v1/20260318
molmospaces-bench-v2/20260325_1
molmospaces-bench-v2/20260327
molmospaces-bench-v2/20260330

I checked both my local extracted benchmark cache and the public Hugging Face benchmark metadata, but I could not find:

ithor/rby1_benchmarks/opening_benchmark

In my extracted molmospaces-bench-v2/20260325_1 resources, the available iTHOR RB-Y1 path I found is instead:

ithor/rby1_bennchmarks/door_opening_benchmark

I also found the following RB-Y1 benchmark paths in the extracted resources:

procthor-objaverse/rby1_bennchmarks/pick_benchmark
procthor-objaverse/rby1_bennchmarks/pnp_benchmark
ithor/rby1_bennchmarks/door_opening_benchmark
procthor-10k/rby1_bennchmarks/door_opening_benchmark

Could you clarify which dataset/resource/version contains:

ithor/rby1_benchmarks/opening_benchmark

or whether this path may have a different name in the public release?

I would also really appreciate it if you could take a look at my previous reply when you have time. I am still trying to correctly reproduce and debug the MolmoBot-RBY1Multitask evaluation on RB-Y1 tasks, especially the Pick and Pick-and-Place behavior. In my current setup, Opening and Door Opening show reasonable nonzero success, but Pick remains at 0% and Pick-and-Place is still much lower than the reported result.

I am looking forward to your clarification on the intended benchmark/resource setup and how I can fix the RB-Y1 MolmoBot evaluation behavior.

Thank you again for your help.

[sjauhri]

Hi @jinyoonok2, thanks for sending the videos!

The Door (77/100) and PnP (23/100) metrics seem correct. PnP reported success in molmobot for rby1 was also 22.5%, and this is close to your results.
The opening benchmark should be at the 20260327 version: "molmospaces-bench-v2/20260327/ithor/rby1_benchmarks/opening_benchmark".

Regarding pick, I will have to check further what success criteria have changed such that it is not reporting success even in the videos where it looks like it succeeded.

[sjauhri]

Ok I think I found the issue. Try rerunning the pick experiments with the code change from this PR: allenai/molmospaces#135

It's a very small change to add a .id in rby1_view.py