add remote teleoperation support for SO101 leader arm.#146
Conversation
abaw
force-pushed
the
feat/remote-teleoperation
branch
from
1675200 to
912aa22
Compare
|
@EverNorif I just fixed pre-commit error. Please help to trigger the check. |
|
Thank you for your contribution. I will review this PR later. |
EverNorif
left a comment
EverNorif
left a comment
There was a problem hiding this comment.
@abaw Great Feature! I have tested the PR and everything works.
I think the documentation could include more detailed installation instructions, and when calibrating the local leader, it could reuse the logic from leisaac.
| - `uv` installed on the local machine (for running the publisher standalone) | ||
| - Network connectivity between the two machines (direct or via SSH tunnel) | ||
| - `pyzmq` installed on the Isaac Sim machine: `pip install leisaac[remote]` or `pip install pyzmq` | ||
|
|
There was a problem hiding this comment.
- I think it would be better to reuse the existing calibration logic in leisaac, so that there is no need to introduce an additional lerobot environment on the local machine.
- Perhaps we can avoid introducing
uvfor now and keep it consistent with the project’s current environment management approach. - It would be better to add a detailed local environment installation guide here; I think you may also need to run
pip install -e source/leisaac[remote]on the local machine?
There was a problem hiding this comment.
These suggestions and questions are related. Let me explain why I used uv in the first place first. I used uv because it supports PEP-508 which let me embed dependencies of the script. And you could run it with uv run xxx.py and all dependencies will be installed in uv's venv. I do this because leisaac has a heavy set of dependencies. And since the local machine does not need any other features of leisaac except forwarding the motor states to the remote. It would be convenient for user to run the script with uv run without pulling a lot of dependencies.
So back to your comments:
-
The dependency of leisaac is avoided on purpose. But I could change it to depend on leisaac if you thought it would better fit the project.
-
uvis only introduced for users to run the script easily. You could use any other tool that supports PEP-508. Or if you already have all dependencies installed, you could just run it directly withoutuv. Souvis not required, it's just my suggested way to run the script. -
No, you don't need to do any
pip installin the local machine.uv runtakes care of all dependencies embedded in the script. But I could improve the doc to make it more clear.
Let me know what you think for 1., I will take your decision.
There was a problem hiding this comment.
I agree that installing a clean environment on the local machine (without Isaac Sim + Lab + torch) is better for joint_state_server.
The pyproject.toml in leisaac does not enforce the installation of Isaac Lab/Sim/torch. I think we can depend on leisaac (to reuse the calibration logic), and at the same time use try(ImportError)-except when importing related packages to skip them and print corresponding warning messages, or perform the imports within the functions where these dependencies are actually used.
Ideally, users should only need to run pip install leisaac[remote] without installing other complex dependencies to run so101_joint_state_server.py.
There was a problem hiding this comment.
Sure, I will try this path. I was assuming Isaac Sim/torch is a dependency of leisaac without confirmation. Thanks for the correction.
| "HF_LEROBOT_CALIBRATION", | ||
| os.path.expanduser("~/.cache/huggingface/lerobot/calibration"), | ||
| ) | ||
| path = os.path.join(base, "teleoperators", "so_leader", f"{arm_id}.json") |
There was a problem hiding this comment.
The cache directories of different versions of lerobot seem to vary. It is recommended to use the cache directory in leisaac.
There was a problem hiding this comment.
can we use the logic of calibrate of SO101Leader in leisaac？
There was a problem hiding this comment.
I have written some related responses here: https://github.com/LightwheelAI/leisaac/pull/146/changes#r3042694594
I could do that if pulling leisaac as a dependency is not an issue.
| @@ -0,0 +1,111 @@ | |||
| # /// script | |||
There was a problem hiding this comment.
Can this script include automatic detection and calibration logic, similar to the teleoperation in leisaac?
There was a problem hiding this comment.
Yes, this might be a better UX instead of relying on another lerobot package.
|
Thanks for the detailed review. I have replied all of them. Let me know what you think on pulling |
|
I am still on this. But I don't have access to a so101 for a while, thus I could not test my changes. It would probably take me some time to get my so101 back. |
Thank you for the progress update. I will test it after you submit the code. :) |
* add SO101LeaderRemote device that receives joint states over ZMQ. * add so101_joint_state_server.py publisher. * add --remote_endpoint flag to teleop_se3_agent.py. * add remote optional dependency (pyzmq) to pyproject.toml. * add remote teleoperation docs.
abaw
force-pushed
the
feat/remote-teleoperation
branch
from
912aa22 to
73b1329
Compare
|
Thanks. I just pushed the code without testing. If you got some time, please help to test it. Otherwise, I could test it after getting my so101 back. |
|
@abaw I have revised some descriptions in the documentation and improved the import methods of the related packages. It now only requires minimal dependencies on the local machine. I have tested it locally, and it works. |
EverNorif
changed the title
2 participants
Closes #144
Summary
Add support for remote teleoperation where the SO101 leader arm is connected to a different machine than the one running Isaac Sim.
Changes
uv run, depends only onfeetech-servo-sdk+pyzmq)Motivation
USB-over-network (VirtualHere) adds ~150ms latency per sync_read vs ~1ms local — unusable at 30Hz. This ZMQ approach achieves 50Hz with negligible overhead.