Integrate with scene editor

[robomotic]

This looks interesting to integrate: https://github.com/ZiYang-xie/WorldGen

[markusgrotz]

I like this idea. It would be similar to GenSim and other works. My concern would be that this includes too many dependency so it would be good to have that in a separate project.

[robomotic]

I will take a deeper look at how it works and see maybe it can be in two stages first we do the splat and then we add the 3d.

[robomotic]

Combining ZiYang-xie/WorldGen and markusgrotz/mujoco-scene-editor allows you to bridge the gap between "Generative 3D Environments" and "Physics-based Robotics Simulation."

While WorldGen can procedurally create a complex 3D world (like a forest, laboratory, or street) from a text prompt, mujoco-scene-editor provides the interactive toolset to turn that visual mesh into a functional MuJoCo simulation environment with robots and physics.

The Integration Workflow

The most effective way to combine them is to use WorldGen as your Environment Asset Generator and mujoco-scene-editor as your Simulation Composer.

1. Generate the Environment (WorldGen)

Use WorldGen to create the high-level geometry. Since MuJoCo works best with meshes, you should use WorldGen's return_mesh=True mode rather than Gaussian Splatting (as splats are not natively collision-aware in physics engines).

Python

from worldgen import WorldGen
import open3d as o3d
# Initialize WorldGen

worldgen = WorldGen(mode="t2s", device="cuda")
# Generate a complex mesh from a prompt

mesh = worldgen.generate_world("A messy futuristic robotic workshop with workbenches and shelves", return_mesh=True)
# Save as PLY (WorldGen's default)

o3d.io.write_triangle_mesh("workshop_env.ply", mesh)

2. Convert to MuJoCo-Compatible Format

MuJoCo 3.x prefers .obj or .stl for meshes. You can use open3d or trimesh to convert the WorldGen output.

Python

# Convert PLY to OBJ (standard for MuJoCo)

mesh_obj = o3d.io.read_triangle_mesh("workshop_env.ply")

o3d.io.write_triangle_mesh("workshop_env.obj", mesh_obj)

3. Assemble and Edit (mujoco-scene-editor)

Now, use the mujoco-scene-editor to import this massive generated mesh and start adding interactive elements.

1. Start the Editor: Run mjcreate from your terminal to open the browser-based editor.

2. Import Asset: Use the "Add Asset" button to load your workshop_env.obj.

3. Define Physics:

   • Set the imported mesh as a static geom (using type="mesh" and contype="1" conaffinity="1" for collisions).

   • Since WorldGen meshes are often single large files, you may want to use a convex decomposition tool (like V-HACD) if you need precise robot-environment interaction, or simply treat it as a visual background with simple invisible boxes for floor/walls.

4. Add Interactive Objects: Use the editor's GUI to place robots (like a Franka Emika or UR5), sensors, and primitive shapes (boxes/spheres) inside the generated environment.

Comparison of Strengths

Feature	WorldGen	mujoco-scene-editor
Role	Creator: Generates the "look" and "geometry" of the world.	Editor: Refines the "logic" and "physics" of the scene.
Input	Text prompts / Images.	MJCF (XML) / Interactive GUI.
Best For	Organic terrain, complex indoor rooms, and artistic variety.	Robot placement, joint tuning, and sensor configuration.

Advanced Idea: Automated "Prompt-to-Sim"

You could write a wrapper script that automates this entire pipeline:

1. Script A: Takes a prompt (e.g., "Mars surface") → Runs WorldGen → Exports .obj.

2. Script B: Generates a template .xml (MJCF) file that includes the new .obj asset.

3. Command: Opens mjedit your_generated_scene.xml.

This allows you to procedurally generate hundreds of unique training environments for Reinforcement Learning (RL) and then use the mujoco-scene-editor to manually verify or "gold-standard" the layouts.

@markusgrotz I am going to try it!