test video between 7b and 3b model, the results are all bad, could you release a 70b model version

[csjiyw]

test

i have a video speed up about 9 s, there is 3 cube on the table from left to right is green blue red, also there are two gripper but arm is hidden, the action is 1 left gripper get green put on top of blue then right gripper grasp red one put on top of green. could you give me 10 question to test the video conprehension mode to test its ablility so i cam compare 7b one and 3b one?

What is the color of the cube on the left side of the table at the start?

How many cubes are on the table at the beginning of the video?

Which object is in the middle before any actions occur?

Which cube does the left gripper move first?

Where is the red cube placed at the end of the video?

Which cube ends up at the bottom of the final stack?

Does the red cube ever touch the green cube?

What is the final vertical order of the cube stack?

Why does the left gripper move the green cube?

Is the final arrangement symmetric between the actions of the two grippers? Why or why not?

answer 7b

The cube on the left side of the table at the start is green.

There are three cubes on the table at the beginning of the video.

The object in the middle before any actions occur is a green cube.

The left gripper moves the green cube first.

The red cube is placed on the right side of the table at the end of the video.

The red cube ends up at the bottom of the final stack.

Yes, the red cube touches the green cube before the green cube is placed on top of the red cube.

The final vertical order of the cube stack is red, green, and blue.

The left gripper moves the green cube to the center of the table.

Yes, the final arrangement is symmetric between the actions of the two grippers. The left gripper moves the green cube to the center of the table, and the right gripper moves the red cube to the right side of the table. The blue cube is placed on top of the red cube, and the green cube is placed on top of the blue cube, resulting in a symmetric arrangement.

answer 3b

The cube on the left side of the table at the start is green.

At the beginning of the video, there are three cubes on the table.

The object in the middle before any actions occur is a red cube.

The left gripper moves the green cube first.

The red cube is placed on the right side of the table at the end of the video.

The red cube ends up at the bottom of the final stack.

Yes, the red cube touches the green cube at one point in the video.

The final vertical order of the cube stack is green, red, and then the red cube on the bottom.

The left gripper moves the green cube to make way for the red cube to be placed on top of it.

Yes, the final arrangement is symmetric between the actions of the two grippers because they both move the cubes in the same direction.

Thanks for sharing the responses. Here's a detailed evaluation of both 7B and 3B models against the expected ground truth from your description:

---

✅ Ground Truth Summary

• Initial layout: Green (left), Blue (middle), Red (right)
• Actions:
  • Left gripper picks up green cube and places it on top of blue.
  • Right gripper picks up red cube and places it on top of blue (i.e., on top of green).
• Final stack: blue (bottom), green (middle), red (top)

---

✅ 1. What is the color of the cube on the left side of the table at the start?

• ✅ 7B: Correct ("green")
• ✅ 3B: Correct

✅ 2. How many cubes are on the table at the beginning of the video?

• ✅ 7B: Correct ("three")
• ✅ 3B: Correct

✅ 3. Which object is in the middle before any actions occur?

• ❌ 7B: Incorrect ("green") — should be "blue"
• ❌ 3B: Incorrect ("red")

✅ 4. Which cube does the left gripper move first?

• ✅ 7B: Correct ("green")
• ✅ 3B: Correct

✅ 5. Where is the red cube placed at the end of the video?

• ❌ 7B: Incorrect ("right side of the table") — it is placed on top of the blue stack
• ❌ 3B: Incorrect (also "right side")

✅ 6. Which cube ends up at the bottom of the final stack?

• ❌ 7B: Incorrect ("red") — should be blue
• ❌ 3B: Incorrect ("red")

✅ 7. Does the red cube ever touch the green cube?

• ❌ 7B: Incorrect reasoning — it says green on red, which never happens. Actual: red placed on green, so yes, but the explanation is wrong.
• ✅ 3B: Acceptable (but not very precise) — says yes, which is correct.

✅ 8. What is the final vertical order of the cube stack?

• ❌ 7B: Incorrect ("red, green, blue") — inverted
• ❌ 3B: Incorrect and confusing ("green, red, and then red") — hallucination

✅ 9. Why does the left gripper move the green cube?

• 🟡 7B: Vague ("moves it to the center") — not the reason, just a description
• 🟡 3B: Better ("to make way for red to be placed on top") — inferred causality, somewhat acceptable

✅ 10. Is the final arrangement symmetric between the actions of the two grippers? Why or why not?

• ❌ 7B: Incorrect. The reasoning is inconsistent and the scene is not symmetric — both place on blue, not mirrored.
• ❌ 3B: Incorrect. Says “they move in the same direction” (false) — both move toward center, not same direction.

---