[AMAZING HAND + ARCTOS] VR Haptic Glove Teleoperation

[danielsanei]

Build a VR-based teleoperation and haptic feedback control system for LeRobot arms.

• Test open-source teleoperation tools (i.e. Phosphobot)
• Design VR glove interface and add haptic feedback (if possible)
• Integrate VR inputs to control robot motion in simulation

[loganli6225]

Project Updates Based On New Hardware

Haptic Glove

LucidVR (https://github.com/LucidVR/lucidgloves)

Robot Arm

Amazing Hand (https://github.com/pollen-robotics/AmazingHand) + ARCTOS

VR Headset

Meta Quest

3D Engine

Unity (https://unity.com/)

No need for LeRobot and Phosphobot anymore

---

New Project Notes (Start with fixed hand so no arm IK)

System Diagram

Meta Quest (OpenXR) → Unity XR Hands (takes data from OpenXR of per-finger joint angles) → ROS2 (takes pose data from Unity through TCP and turns it into ROS2 hand joint messages and publishes) → ROS2 mapper node (subscribes to hand joint messages and maps human finger joints to robot hand joints and publishes – need intermediate step of calibrate, scale, rate-limit) → Simulation or Real (subscribes to robot hand joints with ROS2 controller and updates simulation and publishes force/contact messages) → ROS2 haptics node (subscribes to force/contact messages and maps to LucidVR motor commands) → LucidVR microcontroller takes commands from Serial or Bluetooth

• Meta Quest (OpenXR) --> Unity XR hands: use C# script in Unity
• Unity --> ROS2 MapperNode: publish ros2 topic /vr_hand/joints
• ROS2 MapperNode --> ros2_controller: publish ros2 topic /amazing_hand_controller/commands ros2_controller --> simulation and real: use plugin
• ros2_controller --> rviz: publish ros topic /joint_states
• sim --> ROS2 HapticsNode: publish ros2 topic /amazing_hand/contacts & ros2 topic /amazing_hand/forces
• ROS2 Haptics Node --> LucidVR: send serial msg & publish ros2 topic /haptics/command

---

Meta Quest (OpenXR) → Unity XR Hands

https://developers.meta.com/horizon/documentation/unity/unity-handtracking-overview/?utm_source=chatgpt.com

Unity XR Hands → ROS2

https://github.com/Unity-Technologies/ROS-TCP-Connector and ROS-TCP Endpoint

---

ROS2 Mapper Node Considerations

• Build a per-finger mapping table
• Calibrate real hand’s open hand angles
• Add gain and/or offset for each joint
• Clamp angles, velocity, acceleration to robot limits
• Low-pass filter and/or rate-limit to protect motors
• Publish URDF joint names for simulation

---

Simulation Considerations

• https://github.com/pollen-robotics/AmazingHand?utm_source=chatgpt.com
• Simple contact sensors for fingertips
• Publish contacts (bool) and/or forces (scalar or vector)

---

ROS2 Haptics Node Considerations

• Clamp to safe range
• Scaling for light contacts
• Use Serial or Bluetooth

---

Other Considerations

• Add a logger node that records human joint, robot joint, force, and contact messages
• Add RViz to visualize /tf of fingers, URDF of Amazing Hand, etc. for debugging
• When adding arm later, add an IK node to map wrist + hand pose in space to arm joint positions/trajectories
• Use simulation for visualization right now but later add a Unity visual interface

[loganli6225]

Amazing_Hand_Gripping_Block_Demo_Recording.mov

Road Map of Simulation --> Haptics Side of Project

• XML file containing Amazing Hand model and object/environment (ex. block)
• sim_bridge_ws.py subscribes to a ROS2 message containing motor commands through rosbridge, drives the simulation's motors, and publishes a ROS2 message containing contact forces of fingertips
• haptics_node.py subscribes to a ROS2 message containing contact forces of fingertips and maps forces to LucidVR glove motor commands
• LucidVR glove receives motor commands through serial

Updates

• Using XML file of Amazing Hand from https://github.com/pollen-robotics/AmazingHand/tree/main/Demo
• Added proxy spheres to XML that "glue" onto fingertips for modeling fingertip contact
• Added a fixed test cube to test simulating contact and contact forces between object and proxy fingertips
• Created test script to simulate Amazing Hand grasping a fixed block
• Video shows demo of fingers closing until grasping onto block based on contact between object and proxy fingertips
• Screenshot shows data from demo of raw forces, filtered forces, motor commands, contact states, and number of contacts from timepoints
• Using test script, created first draft of sim_bridge_ws.py that should subscribe to an 'amazing hand joint commands' message through rosbridge, drive simulation, and publish fingertip forces (for haptics_node.py)

Next Steps:

• To continue working on sim_bridge_ws.py: Finish Meta Quest --> Mapper Node side of project so I can connect sim_bridge_ws.py to actual motor commands
• To work on haptics_node.py: Need to do research on how LucidVR glove provides haptics with their motors (ideally, having the actual LucidVR glove to play around with would be useful)

[danielsanei]

Issue

• Infinite Windows restart loop (not recognizing Windows boot manager) on x-11 Dell laptop

Fix

1. Enter BIOS/UEFI (F2 upon restart)
2. Restore "Last Known Good Settings"
3. Reboot PC
4. Use GRUB (Ubuntu bootloader) to select Windows boot manager
5. If Windows encounters errors trying to boot, may need to Reset This PC (Keep My Files)

Notes

• Keep xx-11 laptop for Windows use (xx-2 laptop for Ubuntu use)
• Reset This PC (Keep My Files) --> removes all files except those inside Users/