Skip to content

feat: multi level navigation#2570

Open
aclauer wants to merge 24 commits into
mainfrom
andrew/feat/kronk-nav-4
Open

feat: multi level navigation#2570
aclauer wants to merge 24 commits into
mainfrom
andrew/feat/kronk-nav-4

Conversation

@aclauer

@aclauer aclauer commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Problem

3d navigation and ray tracing integration

Closes DIM-XXX

Solution

Creates a blueprint using ray tracing and mls planner, plus improvements to other modules. Click and point to navigate to goals.

Ray tracer:

  • register point clouds with odometry (breaks on old fastlio recordings!)
  • outputs region bounds for robust updates

Planner:

  • incremental builds of planner artifacts in separate task so ingest isn't blocked
  • path replan on each frame
  • string pulled paths with tunable cost metrics
  • levers for path safety

Other things:

  • add set_motion_mode() to go2 connection, so we can turn off obstacle avoidance and things when instantiating the blueprint
  • add a simple path follower. Replace this with better trajectory controllers for better results.

How to Test

build native modules:

cd ~/dimos   && ( cd dimos/hardware/sensors/lidar/pointlio/cpp && nix build .#pointlio_native )   && ( cd dimos/mapping/ray_tracing/rust && nix build path:. )   && ( cd dimos/navigation/nav_3d/mls_planner/rust && cargo build --release )

run (on jetson on go2)!

dimos --dtop --rerun-host 0.0.0.0 --rerun-open none run unitree-go2-nav-3d

Contributor License Agreement

  • I have read and approved the CLA.

@aclauer aclauer marked this pull request as ready for review June 23, 2026 19:08
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 41.66667% with 168 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/navigation/basic_path_follower/module.py 37.77% 84 Missing ⚠️
dimos/mapping/ray_tracing/test_transformer.py 0.00% 31 Missing ⚠️
.../navigation/nav_3d/mls_planner/test_transformer.py 0.00% 23 Missing ⚠️
dimos/robot/unitree/connection.py 21.42% 11 Missing ⚠️
dimos/robot/unitree/go2/connection.py 30.76% 9 Missing ⚠️
dimos/navigation/nav_3d/mls_planner/goal_relay.py 80.00% 5 Missing ⚠️
...ee/go2/blueprints/navigation/unitree_go2_nav_3d.py 79.16% 5 Missing ⚠️
@@            Coverage Diff             @@
##             main    #2570      +/-   ##
==========================================
+ Coverage   69.61%   70.79%   +1.18%     
==========================================
  Files         878      870       -8     
  Lines       79326    77499    -1827     
  Branches     7126     6884     -242     
==========================================
- Hits        55220    54864     -356     
+ Misses      22301    20833    -1468     
+ Partials     1805     1802       -3     
Flag Coverage Δ
OS-ubuntu-24.04-arm 62.93% <41.66%> (-0.07%) ⬇️
OS-ubuntu-latest 65.75% <41.66%> (-0.08%) ⬇️
Py-3.10 65.75% <41.66%> (-0.08%) ⬇️
Py-3.11 65.75% <41.66%> (-0.07%) ⬇️
Py-3.12 65.75% <41.66%> (-0.08%) ⬇️
Py-3.13 65.74% <41.66%> (-0.08%) ⬇️
Py-3.14 65.76% <41.66%> (-0.08%) ⬇️
Py-3.14t 65.75% <41.66%> (-0.08%) ⬇️
SelfHosted-Large 30.10% <27.67%> (+0.01%) ⬆️
SelfHosted-Linux 37.78% <27.67%> (-0.02%) ⬇️
SelfHosted-macOS 36.57% <27.67%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/mapping/ray_tracing/module.py 100.00% <100.00%> (ø)
...imos/navigation/basic_path_follower/test_module.py 100.00% <100.00%> (ø)
...avigation/nav_3d/mls_planner/mls_planner_native.py 100.00% <100.00%> (ø)
dimos/robot/all_blueprints.py 100.00% <ø> (ø)
.../unitree/go2/blueprints/basic/unitree_go2_basic.py 73.07% <100.00%> (ø)
dimos/navigation/nav_3d/mls_planner/goal_relay.py 80.00% <80.00%> (ø)
...ee/go2/blueprints/navigation/unitree_go2_nav_3d.py 79.16% <79.16%> (ø)
dimos/robot/unitree/go2/connection.py 55.92% <30.76%> (-0.52%) ⬇️
dimos/robot/unitree/connection.py 45.49% <21.42%> (-4.97%) ⬇️
.../navigation/nav_3d/mls_planner/test_transformer.py 12.50% <0.00%> (-1.79%) ⬇️
... and 2 more

... and 14 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@greptile-apps

greptile-apps Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR wires together 3D click-to-navigate on the Unitree Go2: PointLIO provides odometry-registered point clouds, the ray tracer builds a voxel map with incremental region updates, the MLS planner runs string-pulled Dijkstra paths with wall-penalized edges, and a new BasicPathFollower chases a lookahead point to the goal.

  • Planner overhaul: incremental dijkstra_region, multi-candidate start snapping (snap_candidates), truncate_to_safe for cached-path re-validation, and configurable wall-clearance/buffer/step costs.
  • Ray tracer: sensor-frame points are now rotated into world frame via the odometry quaternion before voxel update; batched region bounds are published alongside the local map so the planner can do bounded updates.
  • Robot integration: set_motion_mode added to the Go2 connection; lidar and camera subscriptions are now optional; BasicPathFollower and GoalRelay complete the blueprint.

Confidence Score: 5/5

Safe to merge. The core algorithmic changes are well-tested and the previously flagged robot_search radius and string_pull baseline issues have been correctly fixed.

All three previously identified logic bugs (robot_search radius compared against penalized costs, string_pull stale baseline on infeasible steps, and the stale empty-path guard) are correctly fixed in this revision. The new incremental dijkstra_region, multi-candidate start snapping, truncate_to_safe, and the BasicPathFollower look sound after tracing through the data flow. The remaining comments are non-blocking style and performance suggestions.

No files require special attention for merge safety. dijkstra.rs has a stale comment and ray_tracing/rust/src/main.rs has an unconventional region_bounds encoding, but neither affects correctness.

Important Files Changed

Filename Overview
dimos/navigation/nav_3d/mls_planner/rust/src/planner.rs Core planner logic: multi-candidate start snapping, bounded robot_search with base_cost radius (fix applied), string_pull with correct infinity baseline on infeasible step (fix applied), truncate_to_safe for cached-path revalidation. No blocking issues found.
dimos/navigation/nav_3d/mls_planner/rust/src/dijkstra.rs Adds incremental dijkstra_region and a Weight enum for base/penalized cost selection; Scored tie-break field changed from CellId to VoxelKey but the comment still says 'cell id'; walk_preds gains a seen-set cycle guard.
dimos/mapping/ray_tracing/rust/src/main.rs Sensor-frame points now rotated into world frame using odometry quaternion before map update; region_bounds emitted as PoseStamped with radius/z_min/z_max packed into Quaternion x/y/z fields — non-standard but self-consistent with the planner decoder.
dimos/navigation/basic_path_follower/module.py New module: P-controlled heading toward a lookahead point, path projection for smooth following, identity-checked waypoints clearing on goal reached. Thread safety via RLock looks correct.
dimos/robot/unitree/connection.py Adds set_motion_mode() with CheckMode/SelectMode RTC API calls; replaces print with module-level logger. Mode is checked before switching to avoid an unnecessary sleep(5).
dimos/robot/unitree/go2/blueprints/navigation/unitree_go2_nav_3d.py New blueprint wiring PointLIO → RayTracer → MLSPlanner → GoalRelay → BasicPathFollower with sensible default parameters; obstacle_avoidance=False correctly deferred to the MLS planner.
dimos/navigation/nav_3d/mls_planner/goal_relay.py Simple adapter forwarding odometry and click goals to PoseStamped; straightforward and correct.
dimos/navigation/nav_3d/mls_planner/rust/src/main.rs Worker pattern with stamp-paired local_map + region_bounds, Tokio Notify wake, blocking block_in_place for CPU-heavy ingest, and throttled viz publishing. Logic looks correct.
dimos/mapping/ray_tracing/rust/src/voxel_ray_tracer.rs Adds batch_local_bounds with percentile-capped radius/z bounds; IRLS-based surface normal estimation with parallelised pooled_normal_and_recency. Percentile function handles edge cases correctly.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant HW as Go2 Hardware
    participant PL as PointLIO
    participant RT as RayTracingVoxelMap
    participant MP as MLSPlannerNative
    participant GR as GoalRelay
    participant BF as BasicPathFollower
    participant MM as MovementManager

    HW->>PL: lidar scan + odom
    PL->>RT: local point cloud (body frame)
    RT->>RT: rotate points to world frame
    RT->>MP: local_map (PointCloud2)
    RT->>MP: region_bounds (PoseStamped)
    MP->>MP: "stamps_paired? pair & hand_off"
    MP->>MP: update_region (Dijkstra incremental)
    HW->>GR: odometry
    GR->>MP: start_pose
    HW-->>GR: goal click (PointStamped)
    GR->>MP: goal_pose
    MP->>MP: plan_or_truncate
    MP->>BF: path (Path)
    BF->>BF: lookahead control loop
    BF->>MM: nav_cmd_vel (Twist)
    MM->>HW: velocity command
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant HW as Go2 Hardware
    participant PL as PointLIO
    participant RT as RayTracingVoxelMap
    participant MP as MLSPlannerNative
    participant GR as GoalRelay
    participant BF as BasicPathFollower
    participant MM as MovementManager

    HW->>PL: lidar scan + odom
    PL->>RT: local point cloud (body frame)
    RT->>RT: rotate points to world frame
    RT->>MP: local_map (PointCloud2)
    RT->>MP: region_bounds (PoseStamped)
    MP->>MP: "stamps_paired? pair & hand_off"
    MP->>MP: update_region (Dijkstra incremental)
    HW->>GR: odometry
    GR->>MP: start_pose
    HW-->>GR: goal click (PointStamped)
    GR->>MP: goal_pose
    MP->>MP: plan_or_truncate
    MP->>BF: path (Path)
    BF->>BF: lookahead control loop
    BF->>MM: nav_cmd_vel (Twist)
    MM->>HW: velocity command
Loading

Reviews (7): Last reviewed commit: "Merge branch 'main' into andrew/feat/kro..." | Re-trigger Greptile

Comment thread dimos/robot/unitree/connection.py Outdated
Comment thread dimos/navigation/nav_3d/mls_planner/rust/src/planner.rs
Comment thread dimos/navigation/basic_path_follower/module.py
Comment thread dimos/navigation/nav_3d/mls_planner/rust/src/main.rs
Comment thread dimos/navigation/nav_3d/mls_planner/rust/src/planner.rs Outdated
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jun 23, 2026
@aclauer aclauer marked this pull request as draft June 23, 2026 21:18
@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Jun 23, 2026
@aclauer aclauer marked this pull request as ready for review June 23, 2026 23:35
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jun 24, 2026
@github-actions github-actions Bot added ready-to-merge Required CI checks have passed on this PR and removed ready-to-merge Required CI checks have passed on this PR labels Jun 24, 2026
@github-actions github-actions Bot removed the ready-to-merge Required CI checks have passed on this PR label Jun 24, 2026
@github-actions github-actions Bot added the ready-to-merge Required CI checks have passed on this PR label Jun 24, 2026
{"api_id": 1001, "parameter": {"enable": int(enabled)}},
)

def set_motion_mode(self, name: str) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How come this was introduced? generally we never run in "normal" mode - only way we ever used go2 is in sport mode

@leshy leshy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one question around AI / normal mode switcher

class ConnectionConfig(ModuleConfig):
ip: str = Field(default_factory=lambda m: m["g"].robot_ip)
mode: Go2Mode = Go2Mode.DEFAULT
lidar: bool = True

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can actually turn off lidar when this is false, no need to implement here and can merge, but just noting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-to-merge Required CI checks have passed on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants