Skip to content

Le-soleile/HumanoidLab

BranchesTags

Repository files navigation

HumanoidDexLab

English | 中文

English

HumanoidDexLab is a research-oriented Isaac Sim / Isaac Lab project for Unitree G1 tabletop manipulation experiments with NVIDIA GR00T. It provides the glue code for running a G1 scene, keyboard teleoperation, 43D joint-position control, GR00T N1.7 policy-server inference, teleoperation recording, and conversion to a GR00T/LeRobot-style training dataset.

This repository is not a standalone simulator. Physics, rendering, sensors, robot assets, and the base task runtime come from Isaac Sim and Isaac Lab.

What Is Included

  • Isaac Lab GUI launch scripts for G1 manipulation experiments.
  • A wipe_table scene with a table, cloth, visual oil stain, progress signal, and reset/recording support.
  • Keyboard teleoperation for upper-body, hands, waist, and recorder control.
  • 43D joint-position action/state plumbing for GR00T fine-tuning and inference.
  • HDF5/MP4 teleoperation recording.
  • Offline demo augmentation and conversion scripts for GR00T/LeRobot training format.
  • Optional whole-body control ONNX policy paths for local experiments.

Repository Layout

HumanoidDexLab/
  run_gr00t_n17_gui_in_docker.sh      # Main GUI launcher
  stop_gui.sh                         # Stop the GUI/container run
  run_wipe_offline_aug_pipeline.sh    # Offline augmentation + dataset conversion
  run_wipe_official_mimic_pipeline.sh # Experimental Isaac Lab Mimic pipeline
  scripts/
    run_gr00t_n17_g1_task.py          # Main Isaac Lab task adapter
    convert_wipe_teleop_to_lerobot.py # Raw teleop -> GR00T/LeRobot dataset
    expand_wipe_teleop_mimic_style.py # Offline mimic-style augmentation
    convert_wipe_teleop_to_mimic.py   # Teleop -> Isaac Lab Mimic source format
  policies/                           # Local third-party policy files, not required in Git
  teleop_demos/                       # Local recordings, ignored by Git

Prerequisites

  • Ubuntu workstation with NVIDIA GPU and working NVIDIA container runtime.
  • Isaac Sim / Isaac Lab installed in the container, expected at /workspace/IsaacLab.
  • Docker container name defaults to isaaclab-humanoid-gui.
  • GR00T repository and checkpoints are expected outside this project. In the examples below this path is named GR00T_ROOT.
  • GR00T N1.7 policy server if running learned policy inference.

Third-party Isaac, Unitree, NVIDIA, and GR00T assets/checkpoints are not redistributed by this repository. Download and use them under their original licenses.

Run The GUI

Set project paths first:

export PROJECT_DIR=/path/to/HumanoidDexLab
export GR00T_ROOT=/path/to/Isaac-GR00T

Start the wipe-table scene with GR00T policy enabled:

JOINT43_ACTION_REPEAT=5 \
JOINT43_STEP_LIMIT=0.02 \
JOINT43_POLICY_FILTER_ALPHA=0.25 \
DISABLE_POLICY=0 \
TELEOP=1 \
SCENE_VARIANT=wipe_table \
POLICY_ACTION_MODE=joint43 \
JOINT43_FREEZE_LOWER_BODY=1 \
POLICY_HOST=127.0.0.1 \
POLICY_PORT=5557 \
"${PROJECT_DIR}/run_gr00t_n17_gui_in_docker.sh"

Start in teleoperation-only mode:

DISABLE_POLICY=1 \
TELEOP=1 \
SCENE_VARIANT=wipe_table \
POLICY_ACTION_MODE=joint43 \
JOINT43_FREEZE_LOWER_BODY=1 \
"${PROJECT_DIR}/run_gr00t_n17_gui_in_docker.sh"

Stop the GUI run:

"${PROJECT_DIR}/stop_gui.sh"

Keyboard Controls

  • P: switch between policy control and manual teleoperation.
  • T / I: hand and arm WASD mode.
  • U: waist WASD mode.
  • O: wrist rotation mode.
  • W/A/S/D, Q/E: motion controls for the active mode.
  • K / J: close/open hand.
  • H / Y: switch active hand.
  • R: reset the scene.
  • G: recording control.
  • N: create a new recording while recording.

The main keyboard implementation is in scripts/run_gr00t_n17_g1_task.py, class KeyboardTeleop.

Record Demonstrations

Recordings are written to teleop_demos/ as local HDF5/MP4 files. This directory is ignored by Git.

Example recording launch:

RECORD_DEMOS=1 \
RECORD_IMAGES=1 \
RECORD_MP4=1 \
DISABLE_POLICY=1 \
TELEOP=1 \
SCENE_VARIANT=wipe_table \
POLICY_ACTION_MODE=joint43 \
JOINT43_FREEZE_LOWER_BODY=1 \
SHOW_POV_WINDOW=1 \
"${PROJECT_DIR}/run_gr00t_n17_gui_in_docker.sh"

Convert And Augment Data

Convert local teleoperation data into a GR00T/LeRobot-style dataset with offline augmentation:

DATA_SOURCE_HOST="${PROJECT_DIR}/teleop_demos" \
AUG_MULTIPLIER=10 \
TAG=offline_aug \
RAW_OUTPUT_DIR_HOST="${PROJECT_DIR}/mimic_aug" \
OUTPUT_DATASET_HOST="${GR00T_ROOT}/demo_data/g1_wipe_aug" \
ACTION_SMOOTHING=1 \
ACTION_SMOOTHING_WINDOW=7 \
ACTION_SMOOTHING_PASSES=2 \
"${PROJECT_DIR}/run_wipe_offline_aug_pipeline.sh"

The offline augmentation pipeline is mimic-style data expansion. It does not create new physics-validated Isaac Lab Mimic rollouts.

GR00T Fine-Tuning Notes

The current data path is RGB + language + 43D proprioceptive state/action. Depth is not recorded by default. The modality config used in the companion GR00T repo is expected to be similar to:

${GR00T_ROOT}/examples/g1_wipe_43d_modality.py

For 24 GB GPUs, Adafactor is recommended over Adam for memory stability when training GR00T N1.7 with top language layers and action heads.

License

The source code in this repository is released under the MIT License. See LICENSE.

Third-party assets, model weights, ONNX policies, Isaac Sim, Isaac Lab, Unitree assets, and NVIDIA GR00T checkpoints are governed by their own licenses and are not covered by this repository license.

Status

This is a research prototype. It is useful as a reproducible integration reference for Isaac Lab + G1 + GR00T data collection and fine-tuning, but it should not be treated as a production robot-control stack.

中文

HumanoidDexLab 是一个面向研究和复现实验的 Isaac Sim / Isaac Lab 工程，用于 Unitree G1 桌面操作任务和 NVIDIA GR00T 接入。它主要提供 G1 仿真场景、键盘遥控、43D 关节位置控制、GR00T N1.7 policy server 推理、遥操作数据录制，以及转换成 GR00T/LeRobot 训练数据格式的胶水代码。

这个项目不是完整独立的仿真器。物理、渲染、传感器、机器人资产和基础任务运行时都来自 Isaac Sim / Isaac Lab。

包含内容

  • G1 操作实验的 Isaac Lab GUI 启动脚本。
  • wipe_table 场景：桌子、布料、视觉油污、进度条、重置和录制支持。
  • 上半身、手、腰部和录制控制的键盘遥控。
  • 面向 GR00T 微调和推理的 43D joint-position action/state 链路。
  • HDF5/MP4 遥操作录制。
  • 离线数据增强和 GR00T/LeRobot 训练数据转换脚本。
  • 本地实验用的 whole-body control ONNX policy 路径。

目录结构

HumanoidDexLab/
  run_gr00t_n17_gui_in_docker.sh      # 主 GUI 启动脚本
  stop_gui.sh                         # 停止 GUI/容器运行
  run_wipe_offline_aug_pipeline.sh    # 离线增强 + 数据转换
  run_wipe_official_mimic_pipeline.sh # 实验性的 Isaac Lab Mimic 链路
  scripts/
    run_gr00t_n17_g1_task.py          # 主 Isaac Lab 任务适配脚本
    convert_wipe_teleop_to_lerobot.py # 原始遥操作数据 -> GR00T/LeRobot 数据
    expand_wipe_teleop_mimic_style.py # 离线 mimic-style 增强
    convert_wipe_teleop_to_mimic.py   # 遥操作数据 -> Isaac Lab Mimic 源格式
  policies/                           # 本地第三方 policy 文件，不建议提交 Git
  teleop_demos/                       # 本地采集数据，已被 Git 忽略

环境要求

  • 带 NVIDIA GPU 的 Ubuntu 工作站，并正确配置 NVIDIA container runtime。
  • 容器内安装 Isaac Sim / Isaac Lab，默认路径为 /workspace/IsaacLab。
  • Docker 容器名默认是 isaaclab-humanoid-gui。
  • GR00T 仓库和 checkpoint 放在本项目外部。下面示例中用 GR00T_ROOT 表示这个路径。
  • 如果运行学习到的策略推理，需要先启动 GR00T N1.7 policy server。

本仓库不重新分发 Isaac、Unitree、NVIDIA、GR00T 的资产和 checkpoint。请按它们各自的协议下载和使用。

启动 GUI

先设置项目路径：

export PROJECT_DIR=/path/to/HumanoidDexLab
export GR00T_ROOT=/path/to/Isaac-GR00T

启动带 GR00T policy 的擦桌任务：

JOINT43_ACTION_REPEAT=5 \
JOINT43_STEP_LIMIT=0.02 \
JOINT43_POLICY_FILTER_ALPHA=0.25 \
DISABLE_POLICY=0 \
TELEOP=1 \
SCENE_VARIANT=wipe_table \
POLICY_ACTION_MODE=joint43 \
JOINT43_FREEZE_LOWER_BODY=1 \
POLICY_HOST=127.0.0.1 \
POLICY_PORT=5557 \
"${PROJECT_DIR}/run_gr00t_n17_gui_in_docker.sh"

只启动遥控模式：

DISABLE_POLICY=1 \
TELEOP=1 \
SCENE_VARIANT=wipe_table \
POLICY_ACTION_MODE=joint43 \
JOINT43_FREEZE_LOWER_BODY=1 \
"${PROJECT_DIR}/run_gr00t_n17_gui_in_docker.sh"

停止 GUI：

"${PROJECT_DIR}/stop_gui.sh"

键盘控制

  • P：切换 policy 控制和手动遥控。
  • T / I：手和手臂 WASD 模式。
  • U：腰部 WASD 模式。
  • O：手腕旋转模式。
  • W/A/S/D、Q/E：当前模式下的运动控制。
  • K / J：合手/张手。
  • H / Y：切换当前控制的手。
  • R：重置场景。
  • G：录制控制。
  • N：录制过程中创建新文件。

键盘遥控的主要实现位于 scripts/run_gr00t_n17_g1_task.py 的 KeyboardTeleop 类。

录制数据

遥操作数据会写入 teleop_demos/，格式为本地 HDF5/MP4。这个目录默认不提交到 Git。

示例录制启动命令：

RECORD_DEMOS=1 \
RECORD_IMAGES=1 \
RECORD_MP4=1 \
DISABLE_POLICY=1 \
TELEOP=1 \
SCENE_VARIANT=wipe_table \
POLICY_ACTION_MODE=joint43 \
JOINT43_FREEZE_LOWER_BODY=1 \
SHOW_POV_WINDOW=1 \
"${PROJECT_DIR}/run_gr00t_n17_gui_in_docker.sh"

数据转换和增强

把本地遥操作数据离线增强并转换为 GR00T/LeRobot 风格训练数据：

DATA_SOURCE_HOST="${PROJECT_DIR}/teleop_demos" \
AUG_MULTIPLIER=10 \
TAG=offline_aug \
RAW_OUTPUT_DIR_HOST="${PROJECT_DIR}/mimic_aug" \
OUTPUT_DATASET_HOST="${GR00T_ROOT}/demo_data/g1_wipe_aug" \
ACTION_SMOOTHING=1 \
ACTION_SMOOTHING_WINDOW=7 \
ACTION_SMOOTHING_PASSES=2 \
"${PROJECT_DIR}/run_wipe_offline_aug_pipeline.sh"

这个离线增强是 mimic-style 数据扩展，不等价于官方 Isaac Lab Mimic 重新生成的物理有效新轨迹。

GR00T 微调说明

当前数据链路是 RGB + 语言 + 43D 本体状态/动作。默认没有记录深度图。配套 GR00T 仓库里的 modality config 通常类似：

${GR00T_ROOT}/examples/g1_wipe_43d_modality.py

在 24GB 显存 GPU 上微调 GR00T N1.7 时，建议使用 Adafactor，不建议使用 Adam，否则容易在 optimizer step OOM。

协议

本仓库源代码使用 MIT License，见 LICENSE。

第三方资产、模型权重、ONNX policy、Isaac Sim、Isaac Lab、Unitree 资产和 NVIDIA GR00T checkpoint 受各自原始协议约束，不属于本仓库协议覆盖范围。

当前状态

这是研究原型。它适合作为 Isaac Lab + G1 + GR00T 数据采集和微调链路的可复现参考，但不应被视为生产级机器人控制栈。

About

HumanoidDexLab is a research-oriented Isaac Sim / Isaac Lab project for Unitree G1 tabletop manipulation experiments with NVIDIA GR00T.

Resources

Readme

License

MIT license
Activity

Stars

3 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages