romi-teleop

romi-teleop is a LeRobot-based plug-and-play teleoperation toolkit for Romi Lab. It currently provides robot( (dual)UR3, (dual)UR5e) in simulation(MUJOCO) and real world teleoperated by Gello devices, It also support multi cameras(OpenCV, Realsense).

Wanna contribute? Add your own Robot, sensor, and teleoperator as Lerobot Plugins.

Project Structure

.
├── configs                         # YAML configs for teleoperation, recording, and reset workflows.
├── scripts                         # Unified YAML-driven CLI for teleoperate, record, and reset.
├── lerobot_camera_mujoco           # LeRobot camera plugin for rendering MuJoCo cameras.
├── lerobot_teleoperator_gello      # LeRobot teleoperator plugin for one GELLO leader device.
├── lerobot_teleoperator_bi_gello   # LeRobot teleoperator plugin for two GELLO leader devices.
├── lerobot_robot_ur3               # LeRobot robot plugin for a physical UR3 arm.
├── lerobot_robot_ur5e              # LeRobot robot plugin for a physical UR5e arm.
├── lerobot_robot_bi_ur3            # Bimanual wrapper for two physical UR3 arms.
├── lerobot_robot_bi_ur5e           # Bimanual wrapper for two physical UR5e arms.
├── lerobot_robot_sim_ur3e          # MuJoCo UR3e simulation plugin and bundled UR3 assets.
├── lerobot_robot_sim_bi_ur5e       # MuJoCo bimanual UR5e simulation plugin and bundled MJCF assets.
├── third_party                     # External dependencies tracked as git submodules.
│   └── lerobot                     # Hugging Face LeRobot source checkout.
├── calibration                     # Local calibration files generated by LeRobot devices.
├── install_deps.sh                 # Project dependency installation script.
├── requirements.txt                # Python dependencies and editable local plugin installs.
├── .gitmodules                     # Git submodule definitions.
├── LICENSE
└── README.md

Installation

Clone the repository with submodules:

git clone --recurse-submodules git@github.com:romi-lab/romi-teleop.git
cd romi-teleop

If you already cloned without submodules, initialize them manually:

git submodule update --init --recursive

Create and activate a Python environment. Python 3.12 is recommended because the local LeRobot plugins declare requires-python >= 3.12.

conda create -n romi-teleop python=3.12 -y
conda activate romi-teleop

Install LeRobot, MuJoCo, camera dependencies, and all local ROMI plugins:

bash install_deps.sh

The installer performs editable installs from requirements.txt, including:

• third_party/lerobot
• GELLO and bimanual GELLO teleoperator plugins
• UR3/UR5e real robot plugins
• UR3e and bimanual UR5e MuJoCo simulation plugins
• MuJoCo, RealSense, Dynamixel, and recording dependencies

Run basic sanity checks after installation:

python -c "import lerobot; print(lerobot.__file__)"
python -c "import mujoco, dm_control; print('mujoco ok')"
lerobot-find-port

For real GELLO devices, make sure the Dynamixel USB adapters are visible as /dev/ttyUSB* and that your user has permission to access serial devices.

Run Reset, Teleoperate, and Record

IMPORTANCE:

1. Every Time when starting the Teleoperator or Robot, You MUST delete the calibration file and re-calibrate again!

2. When using the left UR3 arm in the lab, you have to DISABLE the ETHERNET/IP feature, you can find this buttom at (PROGRAM->INSTALLATION->NTHERNET/IP) every time at the beginning.

Teleoperation uses real GELLO leader devices in both simulation and real-world modes, so first make sure the Dynamixel USB serial ports are accessible:

ls /dev/ttyUSB*
sudo chmod 666 /dev/ttyUSB*

The chmod command is temporary and must be repeated after unplugging/replugging the USB adapter or rebooting. For a permanent setup, add your user to the serial device group used by your Linux distribution.

Each robot setup has one YAML config under configs/. The same config can be used for reset, teleoperation, and recording:

configs/ur3_real.yaml       # Real single-arm UR3.
configs/ur5e_real.yaml      # Real single-arm UR5e.
configs/bi_ur3_real.yaml    # Real bimanual UR3.
configs/bi_ur5e_real.yaml   # Real bimanual UR5e.
configs/ur3e_sim.yaml       # Simulated UR3e.
configs/bi_ur5e_sim.yaml    # Simulated bimanual UR5e.

Edit the YAML file for your setup before running. The most common fields are:

robot: ur3e
mode: sim
teleop_port: /dev/ttyUSB0
repo_id: USER/DATASET
single_task: Describe the task here.

For bimanual setups, use left/right ports:

left_teleop_port: /dev/ttyUSB0
right_teleop_port: /dev/ttyUSB1

Reset

Reset moves a real robot to its configured start joints. In MuJoCo, it initializes and sanity-checks a fresh simulated scene at the configured initial joint state.

python scripts/reset.py --config configs/ur3_real.yaml
python scripts/reset.py --config configs/bi_ur3_real.yaml
python scripts/reset.py --config configs/ur3e_sim.yaml
python scripts/reset.py --config configs/bi_ur5e_sim.yaml

Teleoperate

python scripts/teleoperate.py --config configs/ur3_real.yaml
python scripts/teleoperate.py --config configs/bi_ur3_real.yaml
python scripts/teleoperate.py --config configs/ur3e_sim.yaml
python scripts/teleoperate.py --config configs/bi_ur5e_sim.yaml

Record

Before recording, set repo_id, single_task, num_episodes, and episode_time_s in the config.

python scripts/record.py --config configs/ur3_real.yaml
python scripts/record.py --config configs/bi_ur3_real.yaml
python scripts/record.py --config configs/ur3e_sim.yaml
python scripts/record.py --config configs/bi_ur5e_sim.yaml

Command-line arguments override YAML values, so quick changes can be passed directly:

python scripts/record.py --config configs/ur3e_sim.yaml --num-episodes 3

Reference Repo

1. https://github.com/F-Fer/lerobot_ur5e_gello
2. https://github.com/huggingface/lerobot