Unitree G1 Human Pose Shadowing

A real-time human pose estimation and shadowing system for the Unitree G1 humanoid robot arm using Intel RealSense D455 camera.

Overview

This project enables the Unitree G1 robot to mirror human movements in real-time by capturing human arm poses through a RealSense camera and translating them to robot joint commands.

Hardware Requirements

Essential Components

• Unitree G1 Humanoid Robot
• Intel RealSense D455 Camera
• PC with the following specifications:
  • Recommended: NVIDIA RTX 4090, 64GB RAM
  • Minimum: Any CUDA-compatible GPU (slower inference on lower specs)
• Ethernet Cable for PC-G1 communication

Performance Notes

• Higher-end hardware provides faster inference and smoother real-time performance
• Lower specification PCs are supported but may experience reduced frame rates

Software Requirements

System Dependencies

• Ubuntu 22.04 (Required)
• CUDA 11.8 (Required for GPU acceleration)
• cuDNN (Compatible with CUDA 11.8)
• Miniconda or Anaconda
• ROS 2 Humble

Installation & Setup

1. Clone the Repository

git clone https://github.com/shubham-goel/4D-Humans.git
git clone https://github.com/ WeGo-Robotics/G1_Shadowing.git
cd G1_Shadowing
cp * ~/4D-Humans
cd ~/4D-Humans

Besides these files, you also need to download the SMPL model. You will need the neutral model for running the code. Please go to the Google Drive link and download the model. Place the model in ./data/ folder under 4D-Humans

2. Create Conda Environment

conda create --name shadow python=3.10
conda activate shadow

3. Install PyTorch with CUDA Support

# Install PyTorch with CUDA 11.8 support
conda install pytorch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 pytorch-cuda=11.8 -c pytorch -c nvidia

4. Install 4D-Humans Package

# Note: Remove pytorch and torchvision from setup.py before installation
pip install -e .[all]

5. Install Additional Dependencies

# Install fvcore, iopath, and pytorch3d
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda install pytorch3d -c pytorch3d

6. Install Unitree Python SDK

Follow the instructions at the offical Unitree Github to install the SDK.

7. Install ROS 2 RealSense Packages

sudo apt install ros-humble-librealsense2*
sudo apt install ros-humble-realsense2-*

Hardware Setup

Camera Connection

1. Connect the Intel RealSense D455 camera to your PC via USB 3.0+
2. Verify camera detection and functionality

Robot Connection

1. Connect the Unitree G1 to your PC using an Ethernet cable
2. Follow the network configuration guide: Unitree G1 Developer Quick Start

Usage

1. Launch RealSense Camera

ros2 launch realsense2_camera rs_launch.py enable_depth:=false rgb_camera.color_profile:=640x480x30

2. Prepare for Pose Shadowing

• Important: Ensure only one person is visible in the camera frame
• Position the person in a zero pose (neutral standing position) similar to the G1 robot's default stance
• Verify clear visibility and proper lighting
• For the first time the algorithm will take some time because of downloading the HMR2.0 model.

3. Run the Shadowing Algorithm

python3 pose_estimation.py --detector regnety

4. Run the Retargeting Algorithm

python3 g1_retargeting.py

Troubleshooting

Common Issues & Solutions

PyTorch Version Conflicts

• Problem: Algorithm fails due to automatic PyTorch updates
• Solution: Ensure PyTorch version remains at 2.4.1 as specified in installation. If the PyTorch is updated also install compatible torchvision.

GCC Compilation Errors

• Problem: GCC-related compilation errors
• Solution: Install compatible GCC version:

conda install -c conda-forge gcc=12.1.0

Camera Connection Issues

• Verify RealSense camera is properly connected and recognized
• Check USB 3.0+ connection for optimal performance
• Test camera with RealSense Viewer application

Network Configuration Problems

• Follow the complete network setup guide from Unitree documentation
• Ensure proper IP configuration between PC and G1
• Verify Ethernet cable connection and network interface status

Performance Issues

• Monitor GPU memory usage and system resources
• Reduce camera resolution if experiencing lag
• Consider upgrading hardware for better real-time performance

System Requirements Summary

Component	Minimum	Recommended
OS	Ubuntu 22.04	Ubuntu 22.04
GPU	CUDA-compatible	RTX 4090
RAM	16GB	64GB
CUDA	11.8	11.8
Python	3.10	3.10

Contributing

Contributions are welcome! Please ensure all pull requests include:

• Detailed description of changes
• Testing on recommended hardware configuration
• Documentation updates if applicable

License

Please refer to the original 4D-Humans repository for licensing information.

Support

For technical issues:

• Check troubleshooting section above
• Review Unitree G1 developer documentation
• Open an issue with detailed error logs and system specifications

Acknowledgments

• 4D-Humans for pose estimation
• Unitree Robotics for G1 platform
• Intel RealSense for depth sensing technology