Which benchmark version does the leaderboard use?

[Undefinedx-Zhang]

The DATA_TYPE_TO_SOURCE_TO_VERSION in molmo_spaces_constants.py currently pins molmospaces-bench-v1 to 20260210, but we found a newer version 20260408 is available on R2. The two versions differ in all benchmark.json files — task_description was rewritten (e.g. "Pick up some shiny silver kettle" → "pick up the kettle.") and new fields task_horizon_sec/task_type were added.

Which version is the leaderboard currently using — 20260210 or 20260408? And will it be updated to the newer version?

[omarrayyann]

Hello @Undefinedx-Zhang! The latest versions in molmo_spaces_constants.py (20260408 in your molmospaces-bench-v1 case) are the ones used in the leaderboard results.

• MolmoSpaces pick task (MS-Pick) uses a one-word object descriptions (e.g. "pick up the kettle").
• MolmoBot pick task (MS-Pick) uses 1/2/3 words object descriptions (e.g. "pick up the shiny silver kettle").

Both benchmark tasks can be found here ms-bench, mb-bench.

Let me know if that answers your question.

[dingtine]

"Following the code and data updates around May 18th, we observed a significant drop in the success rate for the pick-mini task on the leaderboard for the dreamzero model, falling from 50% to 20%.

The core changes we implemented include reducing the maximum step limit from 500 to 455 and simplifying the prompts to a 'one-word' format, while keeping all other assets consistent.

Could you please make the evaluation scripts used for the leaderboard publicly available? Or are there any additional configurations or specific evaluation environments required to reproduce these results?"

@omarrayyann

[BlGene]

@dingtine
The instructions for running evaluation are located here: https://github.com/allenai/molmospaces/tree/main/molmo_spaces/evaluation

Prompts and step limits were not changed by the commits around May 18th. Did you possibly mean another date?

The changes to the code/benchmark are described here:
https://docs.google.com/document/d/1aRJ_NGWBzdLk3jJ71GvYx-dj1nbATQbDGfSG3V4Iy0g/edit?tab=t.0

Pick-v1/1.1 corresponds to MS-Pick evaluations.

[Undefinedx-Zhang]

We will use the latest version of the molmospaces code to run the molmospaces-bench-v1 version from 20260408. Running these benchmarks shown in the picture, can it align with the current leaderboard results?

[BlGene]

@Undefinedx-Zhang Yes, that looks correct.

[Undefinedx-Zhang]

However, I noticed that in the document at https://docs.google.com/document/d/1aRJ_NGWBzdLk3jJ71GvYx-dj1nbATQbDGfSG3V4Iy0g/edit?tab=t.0, the time for the four tasks (pick, pick and place, close, open) in the ms version are 45, 65, 30, and 30 respectively, and policy_dt_ms is 100ms. But in the benchmark-version-v1 (20260408), it is 30, 45, 30, and 30, and the default value of policy_dt_ms is 66ms. Do I need to override task_horizon_sec and policy_dt_ms?

[dingtine]

The latest April code behaves very differently from the February one. Is there a validation report for this? Have you re-run the benchmarks to verify the results?

[dingtine]

@BlGene @omarrayyann

[BlGene]

@dingtine @Undefinedx-Zhang

Thanks for pointing out the performance differences between the posted benchmark numbers and what you are getting. I'm thinking that this difference most likely comes from the following sources.

1. Longer task_horizon_sec times (the ones specified in the documentation)
2. The original (more lenient) MolmoSpaces success condition (MolmoSpaces version 0939e18 or older)
3. Possibly some other differences. (Note: the deamzero inference code is included in the molmospaces repo.)

Option a) Reproduction of old numbers
If your aim is a reproduction of the old results, you can try to make those adaptations and run again.

Option b) Clean runs (I prefer this.)

Looking forward (post CoRL deadline), our plan is to

1. Re-run the DreamZero policy again and put updated results on the leaderboard. (w/ main, new success conditon)
2. (possibly) change the benchmark task_horizon_sec values to the ones in the benchmark JSONs.

The last point depends on how many of the other submissions have been run with short task_horizon_sec vs. the long ones. For now, if possible, run with the long task_horizon_sec values, and we can cut down in evals.

[dingtine]

@BlGene
Thank you for your valuable suggestions.
We hold your benchmark in high regard for its credibility, which is why we’ve been using it to evaluate both public SOTA models and our own internal models. To ensure the integrity and accuracy of our submission, we have put significant effort into replicating your results.
However, we have encountered a recurring issue: while our results for 'Pick' and 'Close' tasks are consistent with your leaderboard, the results for 'Open' and 'PickPlace' show a significant discrepancy that exceeds the acceptable variance. We have conducted multiple evaluation runs, and the pattern remains consistent.

Given that we have been unable to fully reproduce the results claimed on the leaderboard, we have concerns about whether our own evaluation results are trustworthy, and whether they would be accepted by your team for leaderboard submission.

[BlGene]

@dingtine
Could you provide some more information? Which models are you running, and which numbers are you getting?

[dingtine]

I am working on reproducing the evaluation results of DreamZero on the Molospace-MS benchmark. Below are the specific figures I have obtained:

Our reproduced results for the 'Pick' and 'Close' tasks are consistent with the leaderboard and fall within the acceptable margin of error. However, we have consistently observed significantly lower performance on the 'PickPlace' and 'Open' tasks. Despite conducting multiple evaluation runs, we have been unable to identify the root cause of this discrepancy.

We have ensured that all our evaluations were conducted using the exact same execution environment and V1 version as specified in the official documentation. Despite this, the performance gap on the 'PickPlace' and 'Open' tasks persists. Since we have strictly followed the prescribed protocols, we are currently at a loss as to what might be causing such a significant deviation.

[dingtine]

@BlGene