From fe7a7de5874ffbc4ba290e97aae6a863205dd4f0 Mon Sep 17 00:00:00 2001 From: cc Date: Thu, 4 Jun 2026 13:41:32 -0700 Subject: [PATCH 01/15] spec: openspec init --- .gitignore | 2 + docs/coding-agents/index.md | 1 + docs/development/openspec.md | 102 ++++++++++++++ docs/docs.json | 1 + openspec/config.yaml | 45 ++++++ openspec/schemas/dimos-capability/schema.yaml | 128 ++++++++++++++++++ .../dimos-capability/templates/design.md | 35 +++++ .../dimos-capability/templates/docs.md | 19 +++ .../dimos-capability/templates/proposal.md | 32 +++++ .../dimos-capability/templates/spec.md | 16 +++ .../dimos-capability/templates/tasks.md | 15 ++ 11 files changed, 396 insertions(+) create mode 100644 docs/development/openspec.md create mode 100644 openspec/config.yaml create mode 100644 openspec/schemas/dimos-capability/schema.yaml create mode 100644 openspec/schemas/dimos-capability/templates/design.md create mode 100644 openspec/schemas/dimos-capability/templates/docs.md create mode 100644 openspec/schemas/dimos-capability/templates/proposal.md create mode 100644 openspec/schemas/dimos-capability/templates/spec.md create mode 100644 openspec/schemas/dimos-capability/templates/tasks.md diff --git a/.gitignore b/.gitignore index 42bdddfa45..787163e787 100644 --- a/.gitignore +++ b/.gitignore @@ -63,8 +63,10 @@ yolo11n.pt # symlink one of .envrc.* if you'd like to use .envrc .claude +.opencode/ **/CLAUDE.md .direnv/ +.omo/ /logs diff --git a/docs/coding-agents/index.md b/docs/coding-agents/index.md index ff778ac5cf..5ac7c854a7 100644 --- a/docs/coding-agents/index.md +++ b/docs/coding-agents/index.md @@ -3,6 +3,7 @@ ├── worktrees.md (creating provisioned worktrees with `bin/worktree`) ├── style.md (code style guidelines for dimos) ├── testing.md (docs about writing tests) +├── ../development/openspec.md (OpenSpec behavior-spec workflow) ├── docs (these are docs about writing docs) │   ├── codeblocks.md │   ├── doclinks.md diff --git a/docs/development/openspec.md b/docs/development/openspec.md new file mode 100644 index 0000000000..280eb0f57e --- /dev/null +++ b/docs/development/openspec.md @@ -0,0 +1,102 @@ +# OpenSpec Workflow + +DimOS uses OpenSpec as the checked-in planning layer for behavior changes. OpenSpec artifacts live under `openspec/` and should describe what the system is supposed to do, why it is changing, and how contributors or agents should validate the work. + +## Terminology + +Keep these two meanings separate: + +- **OpenSpec capability spec**: Markdown requirements under `openspec/specs//spec.md`. These describe observable behavior and acceptance scenarios. +- **DimOS Spec**: Python Protocol/RPC contracts in files like `dimos/navigation/navigation_spec.py` or `dimos/manipulation/control/arm_driver_spec.py`. These describe module interfaces for code wiring. + +Use "OpenSpec capability spec" in prose when there is any chance of confusion. + +## Schema + +The project uses the `dimos-capability` schema configured in `openspec/config.yaml`. + +The artifact flow is: + +```text +proposal + ├── specs + ├── design + └── docs + └── tasks +``` + +| Artifact | Purpose | +|---|---| +| `proposal.md` | Intent, scope, affected DimOS surfaces, and capability impact. | +| `specs//spec.md` | Behavior-first requirements and scenarios. | +| `design.md` | Module, stream, blueprint, skill/MCP, safety, and rollout decisions. | +| `docs.md` | Documentation impact and doc validation plan. | +| `tasks.md` | Implementation, docs, verification, and manual QA checklist. | + +## When to create a change + +Create an OpenSpec change when work changes observable behavior, public CLI/API/MCP behavior, robot behavior, hardware/simulation/replay workflows, docs that users rely on, or cross-module architecture. + +Do not create a change for a purely mechanical refactor, typo fix, or internal cleanup unless it changes behavior or needs cross-session planning context. + +## Writing specs + +OpenSpec capability specs are behavior contracts, not implementation plans. + +Good spec content: + +- User- or developer-visible behavior. +- Public CLI/API/MCP tool behavior. +- Stream or message behavior that downstream modules rely on. +- Robot safety constraints and hardware/simulation/replay expectations. +- Scenarios that can be tested or manually verified. + +Avoid in specs: + +- Private class/function names. +- Generated-file mechanics. +- Library choices and wiring details. +- Step-by-step implementation tasks. + +Put those details in `design.md` or `tasks.md`. + +## Capability names + +Prefer behavior-domain names over code names. Useful starting points: + +- `module-system` +- `blueprint-composition` +- `cli-lifecycle` +- `agent-skills-mcp` +- `configuration` +- `navigation-stack` +- `manipulation-stack` +- `hardware-adapters` +- `simulation-replay` +- `documentation-system` + +Add specs progressively as changes need them. Do not try to backfill the whole project at once. + +## Validation + +Use OpenSpec validation before implementation and before archiving: + +```bash skip +openspec schema validate dimos-capability +openspec validate +openspec templates --json +``` + +For documentation changes, also run the relevant doc checks from [Writing Docs](/docs/development/writing_docs.md): + +```bash skip +md-babel-py run +``` + +When a change touches blueprint names, module-level blueprint variables, or module registry inputs, run: + +```bash skip +pytest dimos/robot/test_all_blueprints_generation.py +``` + +Then run focused tests for the changed code and manually QA through the actual surface: CLI command, MCP tool, HTTP API, simulation/replay blueprint, hardware procedure, or library driver. diff --git a/docs/docs.json b/docs/docs.json index 58da2ff6a1..f0064c9ab9 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -144,6 +144,7 @@ "group": "Development", "pages": [ "development/conventions", + "development/openspec", "development/testing", "development/docker", "development/grid_testing", diff --git a/openspec/config.yaml b/openspec/config.yaml new file mode 100644 index 0000000000..62a72bba63 --- /dev/null +++ b/openspec/config.yaml @@ -0,0 +1,45 @@ +schema: dimos-capability + +context: | + DimOS is a robotics operating system for generalist robots. Modules communicate + through typed streams (`In[T]`, `Out[T]`) over LCM, SHM, ROS, DDS, or other + transports. Blueprints compose modules into runnable robot stacks. Skills are + `@skill`-annotated RPC methods exposed to agents and MCP clients. + + Terminology boundary: + - "OpenSpec spec" means a behavior specification under `openspec/specs/`. + - "DimOS Spec" means a Python Protocol/RPC contract in `*_spec.py` files, + usually inheriting `dimos.spec.utils.Spec` and `typing.Protocol`. + Keep these separate. OpenSpec specs describe observable behavior; DimOS Specs + describe code-level module interfaces. + + OpenSpec specs should capture current behavior, user/developer-visible + outcomes, public CLI/API/tool surfaces, robot safety constraints, and testable + scenarios. Put implementation choices, class names, module wiring, generated + registry updates, and rollout details in `design.md` or `tasks.md`. + + Documentation lives in: + - `docs/usage/` for user-facing concepts and APIs. + - `docs/capabilities/` for capability and platform guides. + - `docs/development/` for contributor process. + - `docs/coding-agents/` and `AGENTS.md` for coding-agent guidance. + +rules: + proposal: + - "Identify affected DimOS surfaces: modules, streams, blueprints, CLI, skills/MCP, docs, hardware, simulation, replay, or generated registries." + - Use capability names that match behavior domains, not Python class names. + - Mark hardware safety or public API/CLI changes explicitly. + specs: + - Write behavior-first requirements; avoid implementation detail unless it is externally observable. + - Every requirement must include at least one `#### Scenario:` block with concrete observable outcomes. + - Use "OpenSpec capability spec" when prose might otherwise be confused with DimOS Python `Spec` Protocols. + design: + - Call out DimOS `Spec` Protocols, adapter Protocols, blueprint composition, stream names/types, and skill/MCP exposure when relevant. + - Mention generated files and required regeneration commands, especially `pytest dimos/robot/test_all_blueprints_generation.py` for blueprint registry changes. + - Include hardware/simulation/replay assumptions and safety constraints for robot-facing work. + docs: + - List user-facing docs, contributor docs, coding-agent docs, and AGENTS.md updates required by the change. + - Include documentation validation commands for changed docs, such as `doclinks` and `md-babel-py run ` where applicable. + tasks: + - Include verification tasks for OpenSpec validation, relevant pytest targets, type checks when needed, and manual QA through the user-facing surface. + - Add registry generation tasks when blueprint names, module classes, or generated registry inputs change. diff --git a/openspec/schemas/dimos-capability/schema.yaml b/openspec/schemas/dimos-capability/schema.yaml new file mode 100644 index 0000000000..fedb7964ee --- /dev/null +++ b/openspec/schemas/dimos-capability/schema.yaml @@ -0,0 +1,128 @@ +name: dimos-capability +version: 1 +description: DimOS capability workflow - proposal → specs/design/docs → tasks +artifacts: + - id: proposal + generates: proposal.md + description: DimOS change proposal covering intent, scope, capability impact, and affected robot/software surfaces + template: proposal.md + instruction: | + Create the proposal document that establishes WHY this change is needed and what DimOS behavior it affects. + + Sections: + - **Why**: 1-2 concise paragraphs on the problem or opportunity. Explain why the change matters now. + - **What Changes**: Bullet list of added, modified, or removed behavior. Mark public API/CLI or hardware-safety breaking changes with **BREAKING**. + - **Affected DimOS Surfaces**: Identify modules, streams, blueprints, CLI commands, skills/MCP tools, docs, hardware, simulation, replay, generated registries, or external protocols touched by the change. + - **Capabilities**: Identify which OpenSpec capability specs will be created or modified: + - **New Capabilities**: List behavior domains introduced by the change. Each becomes `specs//spec.md`. Use kebab-case names (for example, `agent-skills-mcp`, `blueprint-composition`, `manipulation-stack`). + - **Modified Capabilities**: List existing `openspec/specs//` entries whose requirements change. Only include spec-level behavior changes, not implementation-only refactors. + - **Impact**: Summarize user/developer impact, compatibility risks, dependency changes, documentation updates, and test/QA scope. + + Keep proposals concise. Do not include line-by-line implementation details; put architecture and rollout decisions in `design.md`. + requires: [] + - id: specs + generates: specs/**/*.md + description: Behavior-first OpenSpec capability delta specifications + template: spec.md + instruction: | + Create OpenSpec capability specs that define WHAT DimOS should do, not how it is implemented. + + Create one delta spec file per capability listed in proposal.md: + - New capabilities: use `specs//spec.md` with the exact kebab-case name from the proposal. + - Modified capabilities: use the existing folder from `openspec/specs//`. + + Use these delta sections as `##` headers: + - **ADDED Requirements**: New externally observable behavior. + - **MODIFIED Requirements**: Changed behavior. Include the full updated requirement block, not a partial patch. + - **REMOVED Requirements**: Deprecated behavior. Include **Reason** and **Migration**. + - **RENAMED Requirements**: Name-only changes. Use FROM:/TO: format. + + Requirement format: + - Use `### Requirement: `. + - Use SHALL/MUST for normative requirements. + - Include at least one `#### Scenario: ` per requirement. Scenario headings MUST use exactly four `#` characters. + - Prefer `- **GIVEN**`, `- **WHEN**`, `- **THEN**`, and `- **AND**` bullets. + - Cover happy path plus meaningful edge/error/safety cases. + + DimOS-specific guidance: + - Specify user/developer-visible behavior, robot outcomes, CLI behavior, skill/MCP tool behavior, stream contracts, safety constraints, and compatibility expectations. + - Avoid Python class names, private module internals, transport implementation choices, and generated-file details unless those details are observable API contracts. + - Use "OpenSpec capability spec" in prose when needed to avoid confusion with DimOS Python `Spec` Protocols. + - If the behavior only changes implementation and not observable requirements, do not create a spec delta. + requires: + - proposal + - id: design + generates: design.md + description: DimOS technical design and architecture decisions + template: design.md + instruction: | + Create the design document that explains HOW the change should be implemented in DimOS. + + Include design.md for cross-module changes, new robot/hardware integration, new public interfaces, new dependencies, safety-sensitive behavior, generated registry changes, or unclear architecture. + + Sections: + - **Context**: Current state, relevant modules/blueprints/docs, and constraints. + - **Goals / Non-Goals**: What the design achieves and explicitly excludes. + - **DimOS Architecture**: Modules, streams, transports, blueprints, RPC/module refs, DimOS `Spec` Protocols, adapter Protocols, skills/MCP exposure, CLI entry points, and generated registries involved. + - **Decisions**: Key choices with rationale and alternatives considered. + - **Safety / Simulation / Replay**: Hardware assumptions, sim/replay behavior, safety constraints, and manual QA surface. + - **Risks / Trade-offs**: Known risks and mitigations. + - **Migration / Rollout**: Compatibility, generated files, docs, and deployment steps. + - **Open Questions**: Outstanding decisions or unknowns. + + Reference proposal.md for intent and specs for behavior. Keep line-by-line work in tasks.md. + requires: + - proposal + - id: docs + generates: docs.md + description: Documentation impact plan for user, contributor, and coding-agent docs + template: docs.md + instruction: | + Create the documentation impact plan for the change. + + Sections: + - **User-Facing Docs**: Updates under `docs/usage/`, `docs/capabilities/`, `docs/platforms/`, or README files. + - **Contributor Docs**: Updates under `docs/development/`. + - **Coding-Agent Docs**: Updates under `docs/coding-agents/` or `AGENTS.md`. + - **Doc Validation**: Commands needed for changed docs, such as `doclinks`, `md-babel-py run `, and `bin/gen-diagrams`. + - **No Docs Needed**: If no docs are needed, explain why. + + Match `docs/development/writing_docs.md`: contributor-only docs belong in `docs/development`; user-facing behavior belongs in `docs/usage` or `docs/capabilities`. + requires: + - proposal + - id: tasks + generates: tasks.md + description: Implementation, validation, docs, and manual-QA checklist + template: tasks.md + instruction: | + Create the implementation checklist. The apply phase parses checkbox format, so every actionable task MUST use `- [ ]`. + + Guidelines: + - Group tasks under numbered `##` headings. + - Each task must be `- [ ] X.Y Task description`. + - Keep tasks small enough to complete in one focused session. + - Order tasks by dependency. + - Include docs and validation tasks from docs.md. + - Include generated registry tasks when blueprints or module registry inputs change. + - Include manual QA through the actual user surface: CLI, TUI, HTTP API, MCP tool, simulation/replay blueprint, hardware procedure, or library driver. + + Typical DimOS validation tasks: + - Run `openspec validate `. + - Run focused pytest targets for changed modules. + - Run `pytest dimos/robot/test_all_blueprints_generation.py` when blueprint registry output may change. + - Run docs validation commands for changed docs. + - Run lints/types when the touched area requires them. + + Reference specs for WHAT, design for HOW, and docs.md for documentation work. + requires: + - specs + - design + - docs +apply: + requires: + - tasks + tracks: tasks.md + instruction: | + Read proposal.md, specs, design.md, docs.md, and tasks.md before editing code. + Work through pending tasks, mark checkboxes complete as they finish, and keep artifacts current when implementation changes the plan. + Verify with OpenSpec validation, focused tests, docs checks, and manual QA through the relevant DimOS surface. diff --git a/openspec/schemas/dimos-capability/templates/design.md b/openspec/schemas/dimos-capability/templates/design.md new file mode 100644 index 0000000000..25031ceb8b --- /dev/null +++ b/openspec/schemas/dimos-capability/templates/design.md @@ -0,0 +1,35 @@ +## Context + + + +## Goals / Non-Goals + +**Goals:** + + +**Non-Goals:** + + +## DimOS Architecture + + + +## Decisions + + + +## Safety / Simulation / Replay + + + +## Risks / Trade-offs + + + +## Migration / Rollout + + + +## Open Questions + + diff --git a/openspec/schemas/dimos-capability/templates/docs.md b/openspec/schemas/dimos-capability/templates/docs.md new file mode 100644 index 0000000000..d274aed653 --- /dev/null +++ b/openspec/schemas/dimos-capability/templates/docs.md @@ -0,0 +1,19 @@ +## User-Facing Docs + + + +## Contributor Docs + + + +## Coding-Agent Docs + + + +## Doc Validation + + + +## No Docs Needed + + diff --git a/openspec/schemas/dimos-capability/templates/proposal.md b/openspec/schemas/dimos-capability/templates/proposal.md new file mode 100644 index 0000000000..98d409e8de --- /dev/null +++ b/openspec/schemas/dimos-capability/templates/proposal.md @@ -0,0 +1,32 @@ +## Why + + + +## What Changes + + + +## Affected DimOS Surfaces + + +- Modules/streams: +- Blueprints/CLI: +- Skills/MCP: +- Hardware/simulation/replay: +- Docs/generated registries: + +## Capabilities + +### New Capabilities + +- ``: + +### Modified Capabilities + +- ``: + +## Impact + + diff --git a/openspec/schemas/dimos-capability/templates/spec.md b/openspec/schemas/dimos-capability/templates/spec.md new file mode 100644 index 0000000000..afc0c1ff58 --- /dev/null +++ b/openspec/schemas/dimos-capability/templates/spec.md @@ -0,0 +1,16 @@ +## ADDED Requirements + +### Requirement: + + +#### Scenario: +- **GIVEN** +- **WHEN** +- **THEN** +- **AND** + + diff --git a/openspec/schemas/dimos-capability/templates/tasks.md b/openspec/schemas/dimos-capability/templates/tasks.md new file mode 100644 index 0000000000..b38fcdfabb --- /dev/null +++ b/openspec/schemas/dimos-capability/templates/tasks.md @@ -0,0 +1,15 @@ +## 1. Implementation + +- [ ] 1.1 +- [ ] 1.2 + +## 2. Documentation + +- [ ] 2.1 + +## 3. Verification + +- [ ] 3.1 Run `openspec validate ` +- [ ] 3.2 Run focused tests for changed code +- [ ] 3.3 Run docs validation commands for changed docs +- [ ] 3.4 Manually QA through the relevant DimOS surface (CLI, MCP, simulation/replay, hardware procedure, HTTP API, or library driver) From 76158b261a9a4c3f0509bc7a218db7cfe1010e44 Mon Sep 17 00:00:00 2001 From: cc Date: Mon, 8 Jun 2026 16:20:39 -0700 Subject: [PATCH 02/15] chore: revert change to doc folder --- docs/coding-agents/index.md | 1 - docs/development/openspec.md | 102 ----------------------------------- docs/docs.json | 1 - 3 files changed, 104 deletions(-) delete mode 100644 docs/development/openspec.md diff --git a/docs/coding-agents/index.md b/docs/coding-agents/index.md index 5ac7c854a7..ff778ac5cf 100644 --- a/docs/coding-agents/index.md +++ b/docs/coding-agents/index.md @@ -3,7 +3,6 @@ ├── worktrees.md (creating provisioned worktrees with `bin/worktree`) ├── style.md (code style guidelines for dimos) ├── testing.md (docs about writing tests) -├── ../development/openspec.md (OpenSpec behavior-spec workflow) ├── docs (these are docs about writing docs) │   ├── codeblocks.md │   ├── doclinks.md diff --git a/docs/development/openspec.md b/docs/development/openspec.md deleted file mode 100644 index 280eb0f57e..0000000000 --- a/docs/development/openspec.md +++ /dev/null @@ -1,102 +0,0 @@ -# OpenSpec Workflow - -DimOS uses OpenSpec as the checked-in planning layer for behavior changes. OpenSpec artifacts live under `openspec/` and should describe what the system is supposed to do, why it is changing, and how contributors or agents should validate the work. - -## Terminology - -Keep these two meanings separate: - -- **OpenSpec capability spec**: Markdown requirements under `openspec/specs//spec.md`. These describe observable behavior and acceptance scenarios. -- **DimOS Spec**: Python Protocol/RPC contracts in files like `dimos/navigation/navigation_spec.py` or `dimos/manipulation/control/arm_driver_spec.py`. These describe module interfaces for code wiring. - -Use "OpenSpec capability spec" in prose when there is any chance of confusion. - -## Schema - -The project uses the `dimos-capability` schema configured in `openspec/config.yaml`. - -The artifact flow is: - -```text -proposal - ├── specs - ├── design - └── docs - └── tasks -``` - -| Artifact | Purpose | -|---|---| -| `proposal.md` | Intent, scope, affected DimOS surfaces, and capability impact. | -| `specs//spec.md` | Behavior-first requirements and scenarios. | -| `design.md` | Module, stream, blueprint, skill/MCP, safety, and rollout decisions. | -| `docs.md` | Documentation impact and doc validation plan. | -| `tasks.md` | Implementation, docs, verification, and manual QA checklist. | - -## When to create a change - -Create an OpenSpec change when work changes observable behavior, public CLI/API/MCP behavior, robot behavior, hardware/simulation/replay workflows, docs that users rely on, or cross-module architecture. - -Do not create a change for a purely mechanical refactor, typo fix, or internal cleanup unless it changes behavior or needs cross-session planning context. - -## Writing specs - -OpenSpec capability specs are behavior contracts, not implementation plans. - -Good spec content: - -- User- or developer-visible behavior. -- Public CLI/API/MCP tool behavior. -- Stream or message behavior that downstream modules rely on. -- Robot safety constraints and hardware/simulation/replay expectations. -- Scenarios that can be tested or manually verified. - -Avoid in specs: - -- Private class/function names. -- Generated-file mechanics. -- Library choices and wiring details. -- Step-by-step implementation tasks. - -Put those details in `design.md` or `tasks.md`. - -## Capability names - -Prefer behavior-domain names over code names. Useful starting points: - -- `module-system` -- `blueprint-composition` -- `cli-lifecycle` -- `agent-skills-mcp` -- `configuration` -- `navigation-stack` -- `manipulation-stack` -- `hardware-adapters` -- `simulation-replay` -- `documentation-system` - -Add specs progressively as changes need them. Do not try to backfill the whole project at once. - -## Validation - -Use OpenSpec validation before implementation and before archiving: - -```bash skip -openspec schema validate dimos-capability -openspec validate -openspec templates --json -``` - -For documentation changes, also run the relevant doc checks from [Writing Docs](/docs/development/writing_docs.md): - -```bash skip -md-babel-py run -``` - -When a change touches blueprint names, module-level blueprint variables, or module registry inputs, run: - -```bash skip -pytest dimos/robot/test_all_blueprints_generation.py -``` - -Then run focused tests for the changed code and manually QA through the actual surface: CLI command, MCP tool, HTTP API, simulation/replay blueprint, hardware procedure, or library driver. diff --git a/docs/docs.json b/docs/docs.json index f0064c9ab9..58da2ff6a1 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -144,7 +144,6 @@ "group": "Development", "pages": [ "development/conventions", - "development/openspec", "development/testing", "development/docker", "development/grid_testing", From 29ab6453f8e7c0614a2b1b2bfa5da2baf6f698c8 Mon Sep 17 00:00:00 2001 From: cc Date: Mon, 15 Jun 2026 21:09:24 -0700 Subject: [PATCH 03/15] feat: add git-backed robot assets --- CONTEXT.md | 37 +++ dimos/control/examples/cartesian_ik_jogger.py | 7 +- .../cartesian_ik_task/cartesian_ik_task.py | 5 +- .../control/tasks/teleop_task/teleop_task.py | 5 +- .../manipulation/planning/utils/mesh_utils.py | 11 +- .../planning/world/drake_world.py | 5 +- .../manipulation/test_manipulation_module.py | 10 +- dimos/robot/asset_manager.py | 282 ++++++++++++++++++ dimos/robot/catalog/a750.py | 11 +- dimos/robot/catalog/piper.py | 12 +- dimos/robot/catalog/ufactory.py | 12 +- dimos/robot/config.py | 8 +- dimos/robot/model_parser.py | 3 +- dimos/robot/robot_asset_declarations.py | 79 +++++ dimos/robot/test_asset_manager.py | 102 +++++++ dimos/utils/ament_prefix.py | 4 +- dimos/utils/git_asset_cache.py | 171 +++++++++++ dimos/utils/test_git_asset_cache.py | 111 +++++++ .../0001-git-backed-robot-asset-manager.md | 7 + docs/capabilities/manipulation/a750.md | 4 +- .../manipulation/adding_a_custom_arm.md | 14 +- docs/capabilities/manipulation/readme.md | 10 + docs/coding-agents/code-quality-rules.md | 2 +- .../.openspec.yaml | 2 + .../design.md | 87 ++++++ .../migrate-robot-arms-to-git-assets/docs.md | 31 ++ .../proposal.md | 39 +++ .../specs/robot-asset-resolution/spec.md | 93 ++++++ .../migrate-robot-arms-to-git-assets/tasks.md | 36 +++ pyproject.toml | 2 + uv.lock | 37 +++ 31 files changed, 1186 insertions(+), 53 deletions(-) create mode 100644 CONTEXT.md create mode 100644 dimos/robot/asset_manager.py create mode 100644 dimos/robot/robot_asset_declarations.py create mode 100644 dimos/robot/test_asset_manager.py create mode 100644 dimos/utils/git_asset_cache.py create mode 100644 dimos/utils/test_git_asset_cache.py create mode 100644 docs/adr/0001-git-backed-robot-asset-manager.md create mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml create mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/design.md create mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/docs.md create mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/proposal.md create mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md create mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/tasks.md diff --git a/CONTEXT.md b/CONTEXT.md new file mode 100644 index 0000000000..58d241d682 --- /dev/null +++ b/CONTEXT.md @@ -0,0 +1,37 @@ +# DimOS Robotics + +DimOS composes robot software from reusable modules and robot-specific descriptions. This context defines the language used when discussing robot model assets and how DimOS consumes them. + +## Language + +**Robot Description Source**: +An upstream repository that contains a robot's URDF, Xacro, MJCF, meshes, and related package files. +_Avoid_: URDF repo, asset repo, model repo + +**Robot Asset Manager**: +A DimOS-facing service that resolves robot description sources into local filesystem paths for use by robot model consumers. +_Avoid_: LFS replacement, description downloader, asset loader + +**Robot Asset Cache**: +The standard user cache directory where DimOS stores fetched robot description source checkouts for reuse across runs. +_Avoid_: Data directory, vendored assets, repository assets + +**Robot Asset Manifest**: +A DimOS-maintained declaration of robot model assets, their robot description sources, default revisions, package roots, and provenance metadata. It may be represented as typed Python objects; it does not imply a YAML/TOML file. +_Avoid_: Registry, asset list, dependency file, YAML manifest + +**ROS Package Root**: +The local directory corresponding to a ROS-style package name, used to resolve `package://...` URIs and `$(find package_name)` expressions in robot model files. +_Avoid_: Package path, asset package, Python package + +**Artifact Role**: +A string key naming a supported robot model asset file or directory kind. Common roles include `urdf`, `mjcf`, `srdf`, and `mesh_dir`; extra role keys such as `urdf_ik` may be used when a robot needs additional files. Strings are the canonical internal representation. +_Avoid_: Parser mode, arbitrary attachment, file purpose + +**DimOS Robot Model Config**: +A DimOS configuration object that names the model paths, package paths, joints, links, and robot-specific metadata needed by planning, control, simulation, or visualization. +_Avoid_: Robot description, URDF config + +**Registered Description Module**: +An importable description entry provided by a third-party robot description registry. +_Avoid_: Robot description source, GitHub repo diff --git a/dimos/control/examples/cartesian_ik_jogger.py b/dimos/control/examples/cartesian_ik_jogger.py index a5ebb75068..a7a67326fe 100644 --- a/dimos/control/examples/cartesian_ik_jogger.py +++ b/dimos/control/examples/cartesian_ik_jogger.py @@ -162,11 +162,10 @@ def clamp(value: float, min_val: float, max_val: float) -> float: def _get_piper_model_path() -> str: - """Get path to Piper MJCF model.""" - from dimos.utils.data import get_data + """Get path to Piper FK model.""" + from dimos.robot.catalog.piper import PIPER_FK_MODEL - piper_path = get_data("piper_description") - return str(piper_path / "mujoco_model" / "piper_no_gripper_description.xml") + return str(PIPER_FK_MODEL) def run_jogger_ui(model_path: str | None = None, ee_joint_id: int = 6) -> None: diff --git a/dimos/control/tasks/cartesian_ik_task/cartesian_ik_task.py b/dimos/control/tasks/cartesian_ik_task/cartesian_ik_task.py index d43e10333d..cf875add98 100644 --- a/dimos/control/tasks/cartesian_ik_task/cartesian_ik_task.py +++ b/dimos/control/tasks/cartesian_ik_task/cartesian_ik_task.py @@ -86,13 +86,12 @@ class CartesianIKTask(BaseControlTask): outputs JointCommandOutput and participates in joint-level arbitration. Example: - >>> from dimos.utils.data import get_data - >>> piper_path = get_data("piper_description") + >>> from dimos.robot.catalog.piper import PIPER_FK_MODEL >>> task = CartesianIKTask( ... name="cartesian_arm", ... config=CartesianIKTaskConfig( ... joint_names=["joint1", "joint2", "joint3", "joint4", "joint5", "joint6"], - ... model_path=piper_path / "mujoco_model" / "piper_no_gripper_description.xml", + ... model_path=PIPER_FK_MODEL, ... ee_joint_id=6, ... priority=10, ... timeout=0.5, diff --git a/dimos/control/tasks/teleop_task/teleop_task.py b/dimos/control/tasks/teleop_task/teleop_task.py index 32301b1f31..b8b96448cc 100644 --- a/dimos/control/tasks/teleop_task/teleop_task.py +++ b/dimos/control/tasks/teleop_task/teleop_task.py @@ -96,13 +96,12 @@ class TeleopIKTask(BaseControlTask): Outputs JointCommandOutput and participates in joint-level arbitration. Example: - >>> from dimos.utils.data import get_data - >>> piper_path = get_data("piper_description") + >>> from dimos.robot.catalog.piper import PIPER_FK_MODEL >>> task = TeleopIKTask( ... name="teleop_arm", ... config=TeleopIKTaskConfig( ... joint_names=["joint1", "joint2", "joint3", "joint4", "joint5", "joint6"], - ... model_path=piper_path / "mujoco_model" / "piper_no_gripper_description.xml", + ... model_path=PIPER_FK_MODEL, ... ee_joint_id=6, ... priority=10, ... timeout=0.5, diff --git a/dimos/manipulation/planning/utils/mesh_utils.py b/dimos/manipulation/planning/utils/mesh_utils.py index 988a4e5e8e..7180c4f896 100644 --- a/dimos/manipulation/planning/utils/mesh_utils.py +++ b/dimos/manipulation/planning/utils/mesh_utils.py @@ -32,6 +32,7 @@ from __future__ import annotations import hashlib +import os from pathlib import Path import re import shutil @@ -72,7 +73,7 @@ def prepare_urdf_for_drake( Returns: Path to the prepared URDF file (may be cached) """ - urdf_path = Path(urdf_path) + urdf_path = Path(os.fspath(urdf_path)) package_paths = package_paths or {} xacro_args = xacro_args or {} @@ -127,7 +128,11 @@ def _generate_cache_key( # Increment this when adding new processing steps (e.g., stripping transmission blocks) processing_version = "v2" - key_data = f"{processing_version}:{urdf_path}:{mtime}:{sorted(package_paths.items())}:{sorted(xacro_args.items())}:{convert_meshes}" + resolved_package_paths = { + package_name: Path(os.fspath(package_path)).resolve() + for package_name, package_path in package_paths.items() + } + key_data = f"{processing_version}:{urdf_path}:{mtime}:{sorted(resolved_package_paths.items())}:{sorted(xacro_args.items())}:{convert_meshes}" return hashlib.md5(key_data.encode()).hexdigest()[:16] @@ -192,7 +197,7 @@ def replace_uri(match: re.Match[str]) -> str: if pkg_name in package_paths: # Ensure absolute path for proper resolution - pkg_path = Path(package_paths[pkg_name]).resolve() + pkg_path = Path(os.fspath(package_paths[pkg_name])).resolve() full_path = pkg_path / rel_path if full_path.exists(): return f"{full_path}{suffix}" diff --git a/dimos/manipulation/planning/world/drake_world.py b/dimos/manipulation/planning/world/drake_world.py index 429e442434..29669a27f1 100644 --- a/dimos/manipulation/planning/world/drake_world.py +++ b/dimos/manipulation/planning/world/drake_world.py @@ -19,6 +19,7 @@ from concurrent.futures import ThreadPoolExecutor from contextlib import contextmanager from dataclasses import dataclass, field +import os from pathlib import Path from threading import RLock, current_thread from typing import TYPE_CHECKING, Any @@ -236,7 +237,7 @@ def add_robot(self, config: RobotModelConfig) -> WorldRobotID: def _load_model(self, config: RobotModelConfig) -> Any: """Load robot model (URDF/xacro/MJCF) and return model instance.""" - original_path = config.model_path.resolve() + original_path = Path(os.fspath(config.model_path)).resolve() if not original_path.exists(): raise FileNotFoundError(f"Robot model not found: {original_path}") @@ -256,7 +257,7 @@ def _load_model(self, config: RobotModelConfig) -> Any: # Register package paths (not applicable to MJCF) if config.package_paths: for pkg_name, pkg_path in config.package_paths.items(): - self._parser.package_map().Add(pkg_name, Path(pkg_path)) + self._parser.package_map().Add(pkg_name, Path(os.fspath(pkg_path))) else: self._parser.package_map().Add( f"{config.name}_description", prepared_path_obj.parent diff --git a/dimos/manipulation/test_manipulation_module.py b/dimos/manipulation/test_manipulation_module.py index 06970258b7..f6999ade24 100644 --- a/dimos/manipulation/test_manipulation_module.py +++ b/dimos/manipulation/test_manipulation_module.py @@ -36,7 +36,7 @@ from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 from dimos.msgs.sensor_msgs.JointState import JointState -from dimos.utils.data import get_data +from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths pytestmark = pytest.mark.self_hosted @@ -47,8 +47,7 @@ def _drake_available() -> bool: def _xarm_urdf_available() -> bool: try: - desc_path = get_data("xarm_description") - model_path = desc_path / "urdf/xarm_device.urdf.xacro" + model_path = RobotAssetPath("xarm7", "urdf") return model_path.exists() except Exception: return False @@ -56,15 +55,14 @@ def _xarm_urdf_available() -> bool: def _get_xarm7_config() -> RobotModelConfig: """Create XArm7 robot config for testing.""" - desc_path = get_data("xarm_description") return RobotModelConfig( name="test_arm", - model_path=desc_path / "urdf/xarm_device.urdf.xacro", + model_path=RobotAssetPath("xarm7", "urdf"), base_pose=PoseStamped(position=Vector3(), orientation=Quaternion()), joint_names=["joint1", "joint2", "joint3", "joint4", "joint5", "joint6", "joint7"], end_effector_link="link7", base_link="link_base", - package_paths={"xarm_description": desc_path}, + package_paths=robot_asset_package_paths("xarm7"), xacro_args={"dof": "7", "limited": "true"}, auto_convert_meshes=True, max_velocity=1.0, diff --git a/dimos/robot/asset_manager.py b/dimos/robot/asset_manager.py new file mode 100644 index 0000000000..7f057dacc3 --- /dev/null +++ b/dimos/robot/asset_manager.py @@ -0,0 +1,282 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Robot model asset declarations and lazy path adapters.""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path + +from dimos.utils.git_asset_cache import GitAssetCache + + +class RobotAssetError(RuntimeError): + """Raised when a robot asset declaration cannot satisfy a request.""" + + +class ArtifactRole(str, Enum): + """Common robot asset artifact roles. + + Strings are canonical internally; this enum is a convenience for common roles. + """ + + URDF = "urdf" + MJCF = "mjcf" + SRDF = "srdf" + MESH_DIR = "mesh_dir" + + +@dataclass(frozen=True) +class RobotAssetDeclaration: + """Typed declaration for one robot model's assets.""" + + model: str + repo_url: str + ref: str + artifacts: Mapping[str, str] + package_roots: Mapping[str, str] = field(default_factory=dict) + xacro_args: Mapping[str, str] = field(default_factory=dict) + source_name: str | None = None + license: str | None = None + + +class RobotAssetManager: + """Resolve robot model artifacts and package roots from declarations.""" + + def __init__( + self, + declarations: Mapping[str, RobotAssetDeclaration] | None = None, + git_cache: GitAssetCache | None = None, + ) -> None: + self._declarations = dict(declarations or {}) + self._git_cache = git_cache or GitAssetCache() + + def get_declaration(self, model: str) -> RobotAssetDeclaration: + try: + return self._declarations[model] + except KeyError as exc: + available = ", ".join(sorted(self._declarations)) or "none" + raise RobotAssetError( + f"Unknown robot asset model {model!r}. Available models: {available}." + ) from exc + + def resolve_artifact(self, model: str, role: str | ArtifactRole) -> Path: + declaration = self.get_declaration(model) + role_key = _role_key(role) + try: + relative_path = declaration.artifacts[role_key] + except KeyError as exc: + available = ", ".join(sorted(declaration.artifacts)) or "none" + raise RobotAssetError( + f"Robot asset model {model!r} does not declare artifact role {role_key!r}. " + f"Available roles: {available}." + ) from exc + + path = self._checkout(declaration) / relative_path + if not path.exists(): + raise RobotAssetError( + f"Declared artifact {role_key!r} for robot asset model {model!r} does not exist: {path}" + ) + return path + + def resolve_package_root(self, model: str, package_name: str) -> Path: + declaration = self.get_declaration(model) + try: + relative_path = declaration.package_roots[package_name] + except KeyError as exc: + available = ", ".join(sorted(declaration.package_roots)) or "none" + raise RobotAssetError( + f"Robot asset model {model!r} does not declare ROS package root " + f"{package_name!r}. Available package roots: {available}." + ) from exc + + path = self._checkout(declaration) / relative_path + if not path.exists(): + raise RobotAssetError( + f"Declared package root {package_name!r} for robot asset model {model!r} " + f"does not exist: {path}" + ) + return path + + def package_roots(self, model: str) -> dict[str, Path]: + declaration = self.get_declaration(model) + return { + package_name: RobotAssetPackagePath(model, package_name, manager=self) + for package_name in declaration.package_roots + } + + def xacro_args(self, model: str) -> dict[str, str]: + return dict(self.get_declaration(model).xacro_args) + + def _checkout(self, declaration: RobotAssetDeclaration) -> Path: + return self._git_cache.resolve(declaration.repo_url, declaration.ref).path + + +class RobotAssetPath(type(Path())): # type: ignore[misc] + """Lazy Path-like adapter for a declared robot model artifact.""" + + def __new__( + cls, + model: str, + role: str | ArtifactRole, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> RobotAssetPath: + instance: RobotAssetPath = super().__new__(cls, ".") + object.__setattr__(instance, "_robot_asset_model", model) + object.__setattr__(instance, "_robot_asset_role", _role_key(role)) + object.__setattr__( + instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) + ) + object.__setattr__( + instance, "_robot_asset_manager", manager or default_robot_asset_manager() + ) + object.__setattr__(instance, "_robot_asset_resolved_cache", None) + return instance + + def __init__( + self, + model: str, + role: str | ArtifactRole, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> None: + # Path subclasses receive the same constructor arguments in __init__ after + # __new__; consume them so pathlib does not see DimOS-specific kwargs. + del model, role, relative_parts, manager + + def _resolve(self) -> Path: + cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") + if cache is None: + manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") + model = object.__getattribute__(self, "_robot_asset_model") + role = object.__getattribute__(self, "_robot_asset_role") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + cache = manager.resolve_artifact(model, role).joinpath(*relative_parts) + object.__setattr__(self, "_robot_asset_resolved_cache", cache) + return cache + + def __getattribute__(self, name: str) -> object: + try: + object.__getattribute__(self, "_robot_asset_model") + except AttributeError: + return object.__getattribute__(self, name) + + if name.startswith("_robot_asset_") or name in {"_resolve"}: + return object.__getattribute__(self, name) + + return getattr(object.__getattribute__(self, "_resolve")(), name) + + def __str__(self) -> str: + return str(self._resolve()) + + def __fspath__(self) -> str: + return str(self._resolve()) + + def __truediv__(self, other: object) -> RobotAssetPath: + model = object.__getattribute__(self, "_robot_asset_model") + role = object.__getattribute__(self, "_robot_asset_role") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + manager = object.__getattribute__(self, "_robot_asset_manager") + return RobotAssetPath(model, role, *relative_parts, other, manager=manager) + + +class RobotAssetPackagePath(type(Path())): # type: ignore[misc] + """Lazy Path-like adapter for a declared ROS package root.""" + + def __new__( + cls, + model: str, + package_name: str, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> RobotAssetPackagePath: + instance: RobotAssetPackagePath = super().__new__(cls, ".") + object.__setattr__(instance, "_robot_asset_model", model) + object.__setattr__(instance, "_robot_asset_package_name", package_name) + object.__setattr__( + instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) + ) + object.__setattr__( + instance, "_robot_asset_manager", manager or default_robot_asset_manager() + ) + object.__setattr__(instance, "_robot_asset_resolved_cache", None) + return instance + + def __init__( + self, + model: str, + package_name: str, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> None: + del model, package_name, relative_parts, manager + + def _resolve(self) -> Path: + cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") + if cache is None: + manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") + model = object.__getattribute__(self, "_robot_asset_model") + package_name = object.__getattribute__(self, "_robot_asset_package_name") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + cache = manager.resolve_package_root(model, package_name).joinpath(*relative_parts) + object.__setattr__(self, "_robot_asset_resolved_cache", cache) + return cache + + def __getattribute__(self, name: str) -> object: + try: + object.__getattribute__(self, "_robot_asset_model") + except AttributeError: + return object.__getattribute__(self, name) + + if name.startswith("_robot_asset_") or name in {"_resolve"}: + return object.__getattribute__(self, name) + + return getattr(object.__getattribute__(self, "_resolve")(), name) + + def __str__(self) -> str: + return str(self._resolve()) + + def __fspath__(self) -> str: + return str(self._resolve()) + + def __truediv__(self, other: object) -> RobotAssetPackagePath: + model = object.__getattribute__(self, "_robot_asset_model") + package_name = object.__getattribute__(self, "_robot_asset_package_name") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + manager = object.__getattribute__(self, "_robot_asset_manager") + return RobotAssetPackagePath(model, package_name, *relative_parts, other, manager=manager) + + +_DEFAULT_MANAGER: RobotAssetManager | None = None + + +def default_robot_asset_manager() -> RobotAssetManager: + global _DEFAULT_MANAGER + if _DEFAULT_MANAGER is None: + from dimos.robot.robot_asset_declarations import ROBOT_ASSETS + + _DEFAULT_MANAGER = RobotAssetManager(ROBOT_ASSETS) + return _DEFAULT_MANAGER + + +def robot_asset_package_paths(model: str) -> dict[str, Path]: + return default_robot_asset_manager().package_roots(model) + + +def _role_key(role: str | ArtifactRole) -> str: + return role.value if isinstance(role, ArtifactRole) else str(role) diff --git a/dimos/robot/catalog/a750.py b/dimos/robot/catalog/a750.py index 94288d128f..2f19ff7146 100644 --- a/dimos/robot/catalog/a750.py +++ b/dimos/robot/catalog/a750.py @@ -19,10 +19,12 @@ import math from typing import Any +from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath -# Pre-built MJCF for Pinocchio FK (xacro not supported by Pinocchio) +# Static no-gripper URDF for Pinocchio FK. The upstream source only publishes the +# full gripper model, so this generated FK variant intentionally stays on LFS. A750_FK_MODEL = LfsPath("a750_description/urdf/a750_rev1_no_gripper.urdf") # A-750 gripper collision exclusions (parallel jaw gripper) @@ -71,7 +73,7 @@ def a750( """ defaults: dict[str, Any] = { "name": name, - "model_path": LfsPath("a750_description") / "urdf/a750_rev1.urdf", + "model_path": RobotAssetPath("a750", "urdf"), "end_effector_link": "gripper_base", "adapter_type": adapter_type, "address": device_path, @@ -79,10 +81,7 @@ def a750( "base_link": "base_link", "home_joints": [0.0, 0.0, -math.radians(90), 0.0, 0.0, 0.0], "base_pose": [0, 0, 0, 0, 0, 0, 1], # base_pose is where the robot sits in the world - "package_paths": { - "a750_description": LfsPath("a750_description"), - "a750_gazebo": LfsPath("a750_description"), - }, + "package_paths": robot_asset_package_paths("a750"), "xacro_args": {}, "auto_convert_meshes": True, "collision_exclusion_pairs": A750_GRIPPER_COLLISION_EXCLUSIONS, diff --git a/dimos/robot/catalog/piper.py b/dimos/robot/catalog/piper.py index 8c9191c68e..af6e351b49 100644 --- a/dimos/robot/catalog/piper.py +++ b/dimos/robot/catalog/piper.py @@ -18,11 +18,12 @@ from typing import Any +from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath -# Pre-built MJCF for Pinocchio FK (xacro not supported by Pinocchio) -PIPER_FK_MODEL = LfsPath("piper_description/mujoco_model/piper_no_gripper_description.xml") +# Static no-gripper URDF for Pinocchio FK (xacro not supported by Pinocchio) +PIPER_FK_MODEL = RobotAssetPath("piper", "urdf_ik") # Simulation model path (MJCF) PIPER_SIM_PATH = LfsPath("piper/scene.xml") @@ -59,7 +60,7 @@ def piper( """ defaults: dict[str, Any] = { "name": name, - "model_path": LfsPath("piper_description") / "urdf/piper_description.xacro", + "model_path": RobotAssetPath("piper", "urdf"), "end_effector_link": "gripper_base", "adapter_type": adapter_type, "address": address, @@ -67,10 +68,7 @@ def piper( "base_link": "base_link", "home_joints": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "base_pose": [0, y_offset, 0, 0, 0, 0, 1], - "package_paths": { - "piper_description": LfsPath("piper_description"), - "piper_gazebo": LfsPath("piper_description"), - }, + "package_paths": robot_asset_package_paths("piper"), "xacro_args": {}, "auto_convert_meshes": True, "collision_exclusion_pairs": PIPER_GRIPPER_COLLISION_EXCLUSIONS, diff --git a/dimos/robot/catalog/ufactory.py b/dimos/robot/catalog/ufactory.py index ddf93185dc..c43a949f8b 100644 --- a/dimos/robot/catalog/ufactory.py +++ b/dimos/robot/catalog/ufactory.py @@ -18,10 +18,12 @@ from typing import Any +from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath -# Pre-built URDFs for Pinocchio FK (xacro not supported by Pinocchio) +# Pre-built URDFs for Pinocchio FK. The upstream xarm_ros2 source provides +# Xacro-only model files, so these generated FK URDFs intentionally stay on LFS. XARM6_FK_MODEL = LfsPath("xarm_description/urdf/xarm6/xarm6.urdf") XARM7_FK_MODEL = LfsPath("xarm_description/urdf/xarm7/xarm7.urdf") @@ -76,7 +78,7 @@ def xarm7( defaults: dict[str, Any] = { "name": name, - "model_path": LfsPath("xarm_description") / "urdf/xarm_device.urdf.xacro", + "model_path": RobotAssetPath("xarm7", "urdf"), "end_effector_link": "link_tcp" if add_gripper else "link7", "adapter_type": adapter_type, "address": address, @@ -84,7 +86,7 @@ def xarm7( "base_link": "link_base", "home_joints": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "base_pose": [x_offset, y_offset, z_offset, 0, 0, 0, 1], - "package_paths": {"xarm_description": LfsPath("xarm_description")}, + "package_paths": robot_asset_package_paths("xarm7"), "xacro_args": xacro_args, "auto_convert_meshes": True, "collision_exclusion_pairs": XARM_GRIPPER_COLLISION_EXCLUSIONS if add_gripper else [], @@ -128,7 +130,7 @@ def xarm6( defaults: dict[str, Any] = { "name": name, - "model_path": LfsPath("xarm_description") / "urdf/xarm_device.urdf.xacro", + "model_path": RobotAssetPath("xarm6", "urdf"), "end_effector_link": "link_tcp" if add_gripper else "link6", "adapter_type": adapter_type, "address": address, @@ -136,7 +138,7 @@ def xarm6( "base_link": "link_base", "home_joints": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "base_pose": [x_offset, y_offset, z_offset, 0, 0, 0, 1], - "package_paths": {"xarm_description": LfsPath("xarm_description")}, + "package_paths": robot_asset_package_paths("xarm6"), "xacro_args": xacro_args, "auto_convert_meshes": True, "collision_exclusion_pairs": XARM_GRIPPER_COLLISION_EXCLUSIONS if add_gripper else [], diff --git a/dimos/robot/config.py b/dimos/robot/config.py index d0922cb842..ae64b31c0d 100644 --- a/dimos/robot/config.py +++ b/dimos/robot/config.py @@ -21,6 +21,7 @@ from __future__ import annotations +import os from pathlib import Path from typing import Any @@ -221,12 +222,15 @@ def to_robot_model_config(self) -> RobotModelConfig: return RobotModelConfig( name=self.name, - model_path=self.model_path, + model_path=Path(os.fspath(self.model_path)), base_pose=base_pose, joint_names=joint_names, end_effector_link=self.end_effector_link, base_link=base_link, - package_paths=self.package_paths, + package_paths={ + package_name: Path(os.fspath(package_path)) + for package_name, package_path in self.package_paths.items() + }, xacro_args=self.xacro_args, collision_exclusion_pairs=exclusions, auto_convert_meshes=self.auto_convert_meshes, diff --git a/dimos/robot/model_parser.py b/dimos/robot/model_parser.py index 1b760d065d..5017738a91 100644 --- a/dimos/robot/model_parser.py +++ b/dimos/robot/model_parser.py @@ -17,6 +17,7 @@ from __future__ import annotations from dataclasses import dataclass, field +import os from pathlib import Path import xml.etree.ElementTree as ET @@ -68,7 +69,7 @@ def parse_model( xacro_args: dict[str, str] | None = None, ) -> ModelDescription: """Parse a robot description file (.urdf, .xacro, .xml/MJCF).""" - path = Path(path) + path = Path(os.fspath(path)) if not path.exists(): raise FileNotFoundError(f"Robot model file not found: {path}") diff --git a/dimos/robot/robot_asset_declarations.py b/dimos/robot/robot_asset_declarations.py new file mode 100644 index 0000000000..cdbf8059ae --- /dev/null +++ b/dimos/robot/robot_asset_declarations.py @@ -0,0 +1,79 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Robot asset declarations resolved by :mod:`dimos.robot.asset_manager`.""" + +from __future__ import annotations + +from dimos.robot.asset_manager import RobotAssetDeclaration + +XARM_ROS2_REPO = "https://github.com/xArm-Developer/xarm_ros2" +PIPER_DESCRIPTION_REPO = "https://github.com/agilexrobotics/agx_arm_urdf" +A750_DESCRIPTION_REPO = "https://github.com/adob/a750_description" + + +ROBOT_ASSETS: dict[str, RobotAssetDeclaration] = { + "xarm6": RobotAssetDeclaration( + model="xarm6", + repo_url=XARM_ROS2_REPO, + ref="humble", + artifacts={ + "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", + "mesh_dir": "xarm_description/meshes", + }, + package_roots={"xarm_description": "xarm_description"}, + xacro_args={"dof": "6", "limited": "true"}, + source_name="xarm_ros2", + ), + "xarm7": RobotAssetDeclaration( + model="xarm7", + repo_url=XARM_ROS2_REPO, + ref="humble", + artifacts={ + "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", + "mesh_dir": "xarm_description/meshes", + }, + package_roots={"xarm_description": "xarm_description"}, + xacro_args={"dof": "7", "limited": "true"}, + source_name="xarm_ros2", + ), + "piper": RobotAssetDeclaration( + model="piper", + repo_url=PIPER_DESCRIPTION_REPO, + ref="main", + artifacts={ + "urdf": "piper/urdf/piper_with_gripper_description.xacro", + "urdf_ik": "piper/urdf/piper_description.urdf", + "mesh_dir": "piper/meshes", + }, + # Upstream URDFs reference package://agx_arm_description/agx_arm_urdf/... + # and expect the checkout directory to be named agx_arm_urdf inside the + # package root. GitAssetCache preserves that checkout directory name. + package_roots={"agx_arm_description": ".."}, + source_name="agx_arm_urdf", + license="MIT", + ), + "a750": RobotAssetDeclaration( + model="a750", + repo_url=A750_DESCRIPTION_REPO, + ref="master", + artifacts={ + "urdf": "urdf/a750_rev1.urdf", + "mesh_dir": "meshes/a750_rev1", + }, + package_roots={"a750_description": "."}, + source_name="a750_description", + license="MIT", + ), +} diff --git a/dimos/robot/test_asset_manager.py b/dimos/robot/test_asset_manager.py new file mode 100644 index 0000000000..d9268beaa9 --- /dev/null +++ b/dimos/robot/test_asset_manager.py @@ -0,0 +1,102 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from pathlib import Path +import subprocess + +import pytest + +from dimos.robot.asset_manager import ( + ArtifactRole, + RobotAssetDeclaration, + RobotAssetError, + RobotAssetManager, + RobotAssetPackagePath, + RobotAssetPath, +) +from dimos.utils.git_asset_cache import GitAssetCache + + +def _git(cwd: Path, *args: str) -> str: + return subprocess.run( + ["git", *args], + cwd=cwd, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +@pytest.fixture() +def asset_manager(tmp_path: Path) -> RobotAssetManager: + repo = tmp_path / "robot_assets" + repo.mkdir() + _git(repo, "init", "-b", "main") + _git(repo, "config", "user.email", "test@example.com") + _git(repo, "config", "user.name", "Test User") + (repo / "robots" / "testbot").mkdir(parents=True) + (repo / "robots" / "testbot" / "model.urdf").write_text("") + (repo / "packages" / "testbot_description").mkdir(parents=True) + (repo / "packages" / "testbot_description" / "package.xml").write_text("") + _git(repo, "add", ".") + _git(repo, "commit", "-m", "assets") + + declaration = RobotAssetDeclaration( + model="testbot", + repo_url=str(repo), + ref="main", + artifacts={"urdf": "robots/testbot/model.urdf"}, + package_roots={"testbot_description": "packages/testbot_description"}, + ) + return RobotAssetManager( + {"testbot": declaration}, + git_cache=GitAssetCache(tmp_path / "cache"), + ) + + +def test_resolves_artifact_paths_and_package_roots(asset_manager: RobotAssetManager) -> None: + artifact = asset_manager.resolve_artifact("testbot", ArtifactRole.URDF) + package_root = asset_manager.resolve_package_root("testbot", "testbot_description") + + assert artifact.name == "model.urdf" + assert artifact.read_text() == "" + assert package_root.name == "testbot_description" + assert (package_root / "package.xml").exists() + + +def test_unknown_model_and_undeclared_artifact_role_raise(asset_manager: RobotAssetManager) -> None: + with pytest.raises(RobotAssetError, match="Unknown robot asset model 'missing'"): + asset_manager.resolve_artifact("missing", ArtifactRole.URDF) + + with pytest.raises(RobotAssetError, match="does not declare artifact role 'mjcf'"): + asset_manager.resolve_artifact("testbot", ArtifactRole.MJCF) + + +def test_lazy_asset_paths_resolve_only_on_path_operations(asset_manager: RobotAssetManager) -> None: + artifact_path = RobotAssetPath("testbot", ArtifactRole.URDF, manager=asset_manager) + package_path = RobotAssetPackagePath("testbot", "testbot_description", manager=asset_manager) + + assert object.__getattribute__(artifact_path, "_robot_asset_resolved_cache") is None + assert object.__getattribute__(package_path, "_robot_asset_resolved_cache") is None + + artifact_string = str(artifact_path) + assert artifact_string.endswith("robots/testbot/model.urdf") + assert object.__getattribute__(artifact_path, "_robot_asset_resolved_cache") is not None + + assert os.fspath(package_path).endswith("packages/testbot_description") + assert object.__getattribute__(package_path, "_robot_asset_resolved_cache") is not None + + assert artifact_path.exists() + assert (package_path / "package.xml").exists() diff --git a/dimos/utils/ament_prefix.py b/dimos/utils/ament_prefix.py index d505a4609b..6118deb61e 100644 --- a/dimos/utils/ament_prefix.py +++ b/dimos/utils/ament_prefix.py @@ -61,7 +61,7 @@ def _setup_ament_index(package_paths: dict[str, Path]) -> None: resource_dir.mkdir(parents=True, exist_ok=True) for pkg_name, pkg_path in package_paths.items(): - resolved = Path(pkg_path).resolve() + resolved = Path(os.fspath(pkg_path)).resolve() if _ament_registered.get(pkg_name) == resolved: continue @@ -95,7 +95,7 @@ def _patch_xacro_find(package_paths: dict[str, Path]) -> Iterator[None]: def custom_find(resolved: str, a: str, args: list[str], context: dict[str, str]) -> str: pkg_name = args[0] if args else "" if pkg_name in package_paths: - pkg_path = str(Path(package_paths[pkg_name]).resolve()) + pkg_path = str(Path(os.fspath(package_paths[pkg_name])).resolve()) return resolved.replace(f"$({a})", pkg_path) return str(original_find(resolved, a, args, context)) diff --git a/dimos/utils/git_asset_cache.py b/dimos/utils/git_asset_cache.py new file mode 100644 index 0000000000..0732c27333 --- /dev/null +++ b/dimos/utils/git_asset_cache.py @@ -0,0 +1,171 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Git-backed cache for robot description sources and other assets.""" + +from __future__ import annotations + +from dataclasses import dataclass +from hashlib import sha256 +from pathlib import Path +import shutil +import tempfile +from urllib.parse import urlparse +import warnings + +from filelock import FileLock +from git import GitCommandError, InvalidGitRepositoryError, NoSuchPathError, Repo + +DEFAULT_ROBOT_ASSET_CACHE_ROOT = Path.home() / ".cache" / "dimos" / "robot_assets" + + +class GitAssetCacheError(RuntimeError): + """Raised when an asset source cannot be resolved.""" + + +class GitAssetCacheWarning(RuntimeWarning): + """Warning emitted when a cached checkout is usable but not fresh.""" + + +@dataclass(frozen=True) +class GitAssetCheckout: + """Resolved local checkout information.""" + + path: Path + repo_url: str + ref: str + updated: bool = False + used_cached_fallback: bool = False + skipped_dirty_update: bool = False + + +class GitAssetCache: + """Resolve `(repo_url, ref)` pairs into fresh-when-safe cached checkouts. + + Policy: + - clone when the cache is missing; + - for clean cached repositories, fetch and check out the declared ref; + - if fetching/updating a cached repository fails, warn and reuse the cache; + - if the cached repository has local changes, warn and skip updates. + """ + + def __init__(self, cache_root: Path | str = DEFAULT_ROBOT_ASSET_CACHE_ROOT) -> None: + self.cache_root = Path(cache_root).expanduser() + self._sources_root = self.cache_root / "sources" + self._locks_root = self.cache_root / "locks" + + def resolve(self, repo_url: str, ref: str) -> GitAssetCheckout: + """Return a local checkout for `repo_url` at `ref`.""" + key = self._source_key(repo_url, ref) + checkout_path = self._sources_root / key / self._repo_slug(repo_url) + lock_path = self._locks_root / f"{key}.lock" + + self._sources_root.mkdir(parents=True, exist_ok=True) + self._locks_root.mkdir(parents=True, exist_ok=True) + + with FileLock(str(lock_path)): + if not checkout_path.exists(): + return self._clone_missing(repo_url, ref, checkout_path) + return self._refresh_cached(repo_url, ref, checkout_path) + + @staticmethod + def _source_key(repo_url: str, ref: str) -> str: + digest = sha256(f"{repo_url}\0{ref}".encode()).hexdigest()[:16] + return digest + + @staticmethod + def _repo_slug(repo_url: str) -> str: + parsed_path = urlparse(repo_url).path or repo_url + slug = Path(parsed_path.rstrip("/")).name + if slug.endswith(".git"): + slug = slug[:-4] + slug = "".join(char if char.isalnum() or char in {"-", "_", "."} else "-" for char in slug) + return slug or "checkout" + + def _clone_missing(self, repo_url: str, ref: str, checkout_path: Path) -> GitAssetCheckout: + temp_parent = checkout_path.parent + temp_parent.mkdir(parents=True, exist_ok=True) + temp_path = Path(tempfile.mkdtemp(prefix=f".{checkout_path.name}-", dir=temp_parent)) + try: + repo = Repo.clone_from(repo_url, temp_path) + self._checkout_ref(repo, ref) + temp_path.rename(checkout_path) + return GitAssetCheckout(path=checkout_path, repo_url=repo_url, ref=ref, updated=True) + except Exception as exc: + shutil.rmtree(temp_path, ignore_errors=True) + raise GitAssetCacheError( + f"Failed to fetch robot asset source {repo_url!r} at ref {ref!r}: {exc}" + ) from exc + + def _refresh_cached(self, repo_url: str, ref: str, checkout_path: Path) -> GitAssetCheckout: + try: + repo = Repo(checkout_path) + except (InvalidGitRepositoryError, NoSuchPathError) as exc: + raise GitAssetCacheError( + f"Cached asset path {checkout_path} is not a valid Git repository" + ) from exc + + if self._is_dirty(repo): + warnings.warn( + f"Robot asset cache {checkout_path} has local changes; skipping upstream update.", + GitAssetCacheWarning, + stacklevel=2, + ) + return GitAssetCheckout( + path=checkout_path, + repo_url=repo_url, + ref=ref, + skipped_dirty_update=True, + ) + + try: + repo.remotes.origin.fetch(tags=True) + before = repo.head.commit.hexsha if repo.head.is_valid() else None + self._checkout_ref(repo, ref) + after = repo.head.commit.hexsha if repo.head.is_valid() else None + return GitAssetCheckout( + path=checkout_path, repo_url=repo_url, ref=ref, updated=before != after + ) + except Exception as exc: + warnings.warn( + f"Could not update robot asset cache {checkout_path}; using cached checkout: {exc}", + GitAssetCacheWarning, + stacklevel=2, + ) + return GitAssetCheckout( + path=checkout_path, + repo_url=repo_url, + ref=ref, + used_cached_fallback=True, + ) + + @staticmethod + def _is_dirty(repo: Repo) -> bool: + return repo.is_dirty(untracked_files=True) + + @staticmethod + def _checkout_ref(repo: Repo, ref: str) -> None: + """Check out a branch, tag, or commit and update clean worktrees safely.""" + remote_ref = f"origin/{ref}" + remote_refs = {str(r) for r in repo.remotes.origin.refs} + + if remote_ref in remote_refs: + repo.git.checkout("-B", ref, remote_ref) + repo.git.reset("--hard", remote_ref) + return + + try: + repo.git.checkout(ref) + except GitCommandError as exc: + raise GitAssetCacheError(f"Could not check out ref {ref!r}") from exc diff --git a/dimos/utils/test_git_asset_cache.py b/dimos/utils/test_git_asset_cache.py new file mode 100644 index 0000000000..4905ed9c3c --- /dev/null +++ b/dimos/utils/test_git_asset_cache.py @@ -0,0 +1,111 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path +import shutil +import subprocess + +import pytest + +from dimos.utils.git_asset_cache import GitAssetCache, GitAssetCacheWarning + + +def _git(cwd: Path, *args: str) -> str: + return subprocess.run( + ["git", *args], + cwd=cwd, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + +def _commit(repo: Path, relative_path: str, contents: str, message: str) -> str: + path = repo / relative_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(contents) + _git(repo, "add", relative_path) + _git(repo, "commit", "-m", message) + return _git(repo, "rev-parse", "HEAD") + + +@pytest.fixture() +def local_repo(tmp_path: Path) -> Path: + repo = tmp_path / "upstream" + repo.mkdir() + _git(repo, "init", "-b", "main") + _git(repo, "config", "user.email", "test@example.com") + _git(repo, "config", "user.name", "Test User") + _commit(repo, "asset.txt", "v1", "initial") + return repo + + +def test_clone_on_miss_resolves_local_git_repo_at_branch(local_repo: Path, tmp_path: Path) -> None: + cache = GitAssetCache(tmp_path / "cache") + + checkout = cache.resolve(str(local_repo), "main") + + assert checkout.updated is True + assert checkout.path.exists() + assert (checkout.path / "asset.txt").read_text() == "v1" + assert _git(checkout.path, "rev-parse", "--abbrev-ref", "HEAD") == "main" + + +def test_clean_cached_repo_updates_when_upstream_branch_changes( + local_repo: Path, tmp_path: Path +) -> None: + cache = GitAssetCache(tmp_path / "cache") + first = cache.resolve(str(local_repo), "main") + first_commit = _git(first.path, "rev-parse", "HEAD") + second_commit = _commit(local_repo, "asset.txt", "v2", "update") + + second = cache.resolve(str(local_repo), "main") + + assert second.path == first.path + assert second.updated is True + assert _git(second.path, "rev-parse", "HEAD") == second_commit + assert _git(second.path, "rev-parse", "HEAD") != first_commit + assert (second.path / "asset.txt").read_text() == "v2" + + +def test_dirty_cached_repo_skips_update_and_preserves_local_edits( + local_repo: Path, tmp_path: Path +) -> None: + cache = GitAssetCache(tmp_path / "cache") + checkout = cache.resolve(str(local_repo), "main") + (checkout.path / "asset.txt").write_text("local edit") + _commit(local_repo, "asset.txt", "upstream edit", "upstream update") + + with pytest.warns(GitAssetCacheWarning, match="local changes"): + dirty_checkout = cache.resolve(str(local_repo), "main") + + assert dirty_checkout.skipped_dirty_update is True + assert (dirty_checkout.path / "asset.txt").read_text() == "local edit" + + +def test_clean_cached_repo_returns_cached_checkout_when_fetch_fails( + local_repo: Path, tmp_path: Path +) -> None: + cache = GitAssetCache(tmp_path / "cache") + checkout = cache.resolve(str(local_repo), "main") + cached_commit = _git(checkout.path, "rev-parse", "HEAD") + shutil.rmtree(local_repo) + + with pytest.warns(GitAssetCacheWarning, match="using cached checkout"): + fallback = cache.resolve(str(local_repo), "main") + + assert fallback.used_cached_fallback is True + assert fallback.path == checkout.path + assert _git(fallback.path, "rev-parse", "HEAD") == cached_commit + assert (fallback.path / "asset.txt").read_text() == "v1" diff --git a/docs/adr/0001-git-backed-robot-asset-manager.md b/docs/adr/0001-git-backed-robot-asset-manager.md new file mode 100644 index 0000000000..b4cb9ab32b --- /dev/null +++ b/docs/adr/0001-git-backed-robot-asset-manager.md @@ -0,0 +1,7 @@ +# Use a Git-backed Robot Asset Manager for robot model files + +DimOS will replace selected Git LFS robot description bundles with a Git-backed Robot Asset Manager that resolves robot model files from upstream robot description sources into a standard user cache. The design prioritizes ease of use and avoiding copied asset bundles in the DimOS repo: model assets are declared robot-first with typed Python objects, use branch/tag/commit refs, deduplicate checkouts by source, update clean cached repos when upstream changes are available, warn and continue with cache on update failure, and skip updates when local cache changes are present. + +We will build this as a thin DimOS layer over existing Git tooling rather than writing Git operations from scratch or depending on `robot_descriptions.py` as the primary abstraction. The robot asset layer exposes flat artifact keys such as `urdf`, `mjcf`, `srdf`, `mesh_dir`, and additional string keys when needed; Xacro files remain ordinary resolved artifacts and are processed by the existing model parsing and Drake preparation layers using declared ROS package roots and xacro arguments. + +This accepts less strict reproducibility by default than commit-only pinning, but keeps commit refs available for cases that need them while making the common path simple and fresh against upstream robot description sources. diff --git a/docs/capabilities/manipulation/a750.md b/docs/capabilities/manipulation/a750.md index aa435bdd85..52e8efc291 100644 --- a/docs/capabilities/manipulation/a750.md +++ b/docs/capabilities/manipulation/a750.md @@ -59,10 +59,10 @@ The robot catalog entry is defined in [`dimos/robot/catalog/a750.py`](/dimos/rob | Base link | `base_link` | | End-effector link | `gripper_base` | | Home joints | `[0, 0, -90 deg, 0, 0, 0]` | -| Drake model | `a750_description/urdf/a750_rev1.urdf` | +| Drake model | `RobotAssetPath("a750", "urdf")` → `urdf/a750_rev1.urdf` from `https://github.com/adob/a750_description` | | FK/keyboard model | `a750_description/urdf/a750_rev1_no_gripper.urdf` | -The no-gripper model is used for Pinocchio FK in keyboard teleop because Pinocchio does not consume the xacro path used by the full robot description. +The runtime model and `a750_description` package root are resolved through the Robot Asset Manager cache. The no-gripper model remains LFS-backed for Pinocchio FK because the upstream source publishes the full gripper URDF but not the generated no-gripper variant used by keyboard teleop. ## Gripper diff --git a/docs/capabilities/manipulation/adding_a_custom_arm.md b/docs/capabilities/manipulation/adding_a_custom_arm.md index 1e0a14a2a0..31feb7c250 100644 --- a/docs/capabilities/manipulation/adding_a_custom_arm.md +++ b/docs/capabilities/manipulation/adding_a_custom_arm.md @@ -503,19 +503,21 @@ If you want motion planning (collision-free trajectories via Drake), you need a ### 4a. Add your URDF -Place your URDF/xacro files under LFS data so they can be resolved via `LfsPath`. `LfsPath` is a `Path` subclass that lazily downloads LFS data on first access — this avoids downloading at import time when the blueprint module is loaded. +Prefer an upstream Robot Description Source and add a typed declaration in `dimos/robot/robot_asset_declarations.py`. `RobotAssetPath` is a lazy `Path`-like adapter: importing the catalog does not clone or update the repo, but the first concrete path access resolves the source into `~/.cache/dimos/robot_assets`. + +Use `LfsPath` only when the asset is intentionally vendored, locally modified, or has no suitable upstream source. ```python skip -from dimos.utils.data import LfsPath +from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths from dimos.manipulation.manipulation_module import manipulation_module from dimos.manipulation.planning.spec import RobotModelConfig from dimos.msgs.geometry_msgs.PoseStamped import PoseStamped from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 -# LfsPath defers download until the path is actually accessed -_YOURARM_URDF_PATH = LfsPath("yourarm_description/urdf/yourarm.urdf") -_YOURARM_PACKAGE_PATH = LfsPath("yourarm_description") +# Add a RobotAssetDeclaration for "yourarm" in robot_asset_declarations.py. +# Common artifact roles are "urdf", "mjcf", "srdf", and "mesh_dir". +_YOURARM_URDF_PATH = RobotAssetPath("yourarm", "urdf") def _make_base_pose(x=0.0, y=0.0, z=0.0) -> PoseStamped: @@ -554,7 +556,7 @@ def _make_yourarm_config( joint_names=joint_names, end_effector_link="link6", # Last link in your URDF's kinematic chain base_link="base_link", # Root link of your URDF - package_paths={"yourarm_description": _YOURARM_PACKAGE_PATH}, + package_paths=robot_asset_package_paths("yourarm"), xacro_args={}, # Xacro arguments if using .xacro files collision_exclusion_pairs=[], # Pairs of links that can touch (e.g., gripper fingers) auto_convert_meshes=True, # Convert DAE/STL meshes for Drake diff --git a/docs/capabilities/manipulation/readme.md b/docs/capabilities/manipulation/readme.md index c9cc01cd42..4c4f258ab5 100644 --- a/docs/capabilities/manipulation/readme.md +++ b/docs/capabilities/manipulation/readme.md @@ -110,6 +110,16 @@ visualization backend. | XArm6 | 6 | Y | Y | — | | XArm7 | 7 | Y | Y | Y | +## Robot model assets + +XArm6, XArm7, Piper, and A-750 runtime model paths are resolved through the Robot Asset Manager (`dimos.robot.asset_manager`). Catalogs use `RobotAssetPath` so imports stay lightweight: no network or Git work happens until a concrete path is accessed. + +Declared upstream sources are cached under `~/.cache/dimos/robot_assets`. The cache is fresh-when-safe: a missing checkout is cloned, a clean checkout is updated, update failures warn and continue with the cached copy, and dirty local checkouts are preserved with a warning. + +Common artifact roles are flat strings: `urdf`, `mjcf`, `srdf`, and `mesh_dir`. Extra roles such as `urdf_ik` may be declared for robot-specific planning/FK needs. + +Some FK-only Pinocchio assets remain on LFS when the upstream source does not publish a static no-gripper URDF/MJCF equivalent. OpenArm also remains on the existing path because DimOS carries local model modifications. + ## Adding a Custom Arm [guide is here](/docs/capabilities/manipulation/adding_a_custom_arm.md) diff --git a/docs/coding-agents/code-quality-rules.md b/docs/coding-agents/code-quality-rules.md index 5b252fc968..67bee4ce95 100644 --- a/docs/coding-agents/code-quality-rules.md +++ b/docs/coding-agents/code-quality-rules.md @@ -16,7 +16,7 @@ Rules dimos code is expected to follow. They address recurring issues found in c * Specify only what differs from defaults. Don't restate defaults like `tick_rate=100.0`, `publish_joint_state=True`, or default topics (`/cmd_vel`, `/odom`). * `.transports({...})` applies to all matching modules, so define a remap once, not twice across sub-blueprints. * No lambdas -- they can't be pickled to worker processes. Use named functions. -* Do no work at import time: no subprocesses, viewers, model parsing, or network. In particular don't call `get_data(...)` (it blocks import until the download finishes) -- use `LfsPath` (resolved at access time) or build the config in `start`/`build`. Any process you start must be managed (shut down when not needed). +* Do no work at import time: no subprocesses, viewers, model parsing, or network. In particular don't call `get_data(...)` (it blocks import until the download finishes) -- use lazy path adapters such as `RobotAssetPath` for upstream robot descriptions or `LfsPath` for intentionally vendored assets, or build the config in `start`/`build`. Any process you start must be managed (shut down when not needed). * Blueprint files define blueprints, not modules/classes. * Helper blueprints not meant to run alone must start with `_` (the `all_blueprints.py` generator skips them); demo/non-shared ones get a `demo_` prefix (hidden from `dimos list`). diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml b/openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml new file mode 100644 index 0000000000..40f4fff67f --- /dev/null +++ b/openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml @@ -0,0 +1,2 @@ +schema: dimos-capability +created: 2026-06-16 diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/design.md b/openspec/changes/migrate-robot-arms-to-git-assets/design.md new file mode 100644 index 0000000000..b6827ee5db --- /dev/null +++ b/openspec/changes/migrate-robot-arms-to-git-assets/design.md @@ -0,0 +1,87 @@ +## Context + +DimOS manipulation robot catalogs currently declare model assets through `LfsPath` values that unpack archives from `data/.lfs`. The affected surfaces include `dimos/robot/catalog/ufactory.py`, `dimos/robot/catalog/piper.py`, `dimos/robot/catalog/a750.py`, manipulator blueprints, teleop/control blueprint wiring, `RobotConfig`, model parsing, Drake preprocessing, and manipulation docs/tests. + +The design decisions captured during exploration are recorded in `docs/adr/0001-git-backed-robot-asset-manager.md` and the glossary in `CONTEXT.md`. The change should migrate xArm, Piper, and A750, while leaving OpenArm on its current path because its description has local DimOS modifications. + +## Goals / Non-Goals + +**Goals:** + +- Resolve xArm, Piper, and A750 model artifacts from upstream robot description sources instead of copied LFS bundles. +- Preserve existing catalog ergonomics: consumers should still receive `Path`-compatible model paths and `dict[str, Path]` package roots. +- Avoid network activity at import time by using a lazy Path-like adapter for catalog constants. +- Support branch, tag, and commit refs, with commit pinning available but not mandatory. +- Use a standard user cache such as `~/.cache/dimos/robot_assets`. +- Update clean cached checkouts when upstream changes are available; warn and use cache when update fails; warn and skip updates for dirty cached checkouts. +- Keep Xacro processing and Drake URDF preparation in the existing parser/preparation layers. + +**Non-Goals:** + +- Do not migrate Unitree robot descriptions in this change. +- Do not migrate OpenArm in this change. +- Do not add local override machinery in v1; local tests can pass explicit `Path` values where needed. +- Do not introduce a YAML/TOML manifest requirement; typed Python declarations are the initial declaration format. +- Do not change CLI, skill/MCP, stream, or hardware command behavior. + +## DimOS Architecture + +Add a small asset layer under the DimOS runtime codebase, with two responsibilities: + +1. A generic Git cache component that resolves `(repo_url, ref)` into a local checkout path. +2. A robot-facing manager that resolves `(robot_model_name, artifact_role)` and ROS package roots from typed Python robot asset declarations. + +The core concepts are: + +- `GitAssetCache`: wraps Git operations and file locking. It clones missing repositories, checks whether cached repositories are dirty, fetches/checks out refs for clean repositories, and emits warnings/fallback reasons for update failures or dirty cache skips. +- `RobotAssetManager`: owns typed robot asset declarations and resolves artifact roles such as `urdf`, `mjcf`, `srdf`, `mesh_dir`, and extra string roles such as `urdf_ik`. +- `RobotAssetPath`: a lazy Path-like adapter used by catalog modules. It should not touch the network or filesystem at import time; it resolves only when a path operation requires a concrete filesystem path. +- Robot asset declarations: typed Python objects keyed by robot model name, containing a robot description source URL/ref, flat `artifacts: dict[str, str]`, `package_roots: dict[str, str]`, optional model-level `xacro_args`, and lightweight provenance fields such as source name/license when known. + +No new DimOS `Spec` Protocol is required unless implementation discovers an RPC/module boundary. This is a local library/API layer used by catalogs and planning/control consumers, not a stream transport or module interface. Existing blueprint composition should continue to consume catalog constants. No skills/MCP exposure or CLI entry point is required for v1. + +Package root handling must preserve existing parser behavior: declared ROS package roots are passed as `dict[str, Path]` to consumers that resolve `package://...` URIs and `$(find package_name)` expressions. Xacro files remain ordinary artifact paths; `dimos/robot/model_parser.py` and Drake preparation utilities remain responsible for Xacro expansion and mesh/package handling. + +Dependencies: add `GitPython` for Git operations and `filelock` for cross-process cache locking. Keep direct shelling out to Git behind the library only if GitPython cannot cover a required operation. + +## Decisions + +- Use a Git-backed Robot Asset Manager instead of Git LFS bundles for xArm, Piper, and A750. This removes copied upstream robot description bundles from the common DimOS maintenance path. +- Use a thin DimOS wrapper over GitPython and filelock rather than `robot_descriptions.py`. `robot_descriptions.py` is useful as a curated registry, but this change needs arbitrary upstream robot description sources and DimOS-specific freshness policy. +- Use robot-model-first declarations. Catalogs should ask for assets by robot model name and artifact role; the manager deduplicates source checkouts internally by `(repo_url, ref)`. +- Use flat string artifact roles. Common role constants may exist for `urdf`, `mjcf`, `srdf`, and `mesh_dir`, but strings are canonical internally and extra roles remain possible. +- Allow branch, tag, and commit refs. Branch/tag defaults optimize ease of use and freshness; commit refs remain available for CI, releases, or fragile assets. +- Use “fresh-when-safe” cache behavior: clone missing cache, update clean cached repositories, warn/use cache on update failure, and warn/skip update for dirty cached repositories. +- Keep Xacro processing outside asset resolution. Asset resolution returns paths and package roots; existing model parsing and Drake preparation layers expand Xacro and normalize URDF/mesh details. + +## Safety / Simulation / Replay + +This change must not alter robot commands, control loops, skills, or stream contracts. Safety risk is indirect: an incorrect model asset can affect planning, FK/IK, visualization, or simulation. Implementation should verify resolved model paths for xArm, Piper, and A750 through existing parsing/planning entry points before using them in robot-facing blueprints. + +Simulation and replay behavior should match current behavior after cache population. First-run network failures should fail clearly when no cached checkout exists. If a cached checkout exists and an upstream update check fails, DimOS should warn and continue with the cached copy. + +Manual QA should cover at least one xArm blueprint path and one non-xArm migrated arm path where practical, using simulation/replay or parser-level checks before any real hardware run. + +## Risks / Trade-offs + +- Upstream repository layouts may differ from current LFS bundle layouts. Mitigation: declare artifact paths and package roots explicitly per robot model and test them. +- Branch/tag refs reduce strict reproducibility compared with commit-only pinning. Mitigation: allow commit refs and document when to pin. +- Freshness checks add network dependency to first resolution. Mitigation: continue with existing cache when update fails, and fail fast only when no cache exists. +- Dirty cache preservation may leave developers on locally modified assets. Mitigation: warn clearly and never overwrite local cache edits automatically. +- Lazy Path behavior can expose edge cases if consumers expect concrete `pathlib.Path` internals. Mitigation: keep the adapter small, test common path operations, and cast to concrete paths at integration boundaries if needed. + +## Migration / Rollout + +1. Add Git cache and robot asset manager code with tests. +2. Add typed asset declarations for xArm, Piper, and A750. +3. Migrate catalog constants and `RobotConfig.package_paths` declarations for xArm first, then Piper and A750. +4. Update tests and docs that mention `LfsPath` as the canonical manipulator asset pattern. +5. Keep existing LFS archives until migrated paths are verified and rollback is no longer needed. +6. If any catalog exports or blueprint registry behavior changes, run `pytest dimos/robot/test_all_blueprints_generation.py`; otherwise no generated blueprint registry update is expected. + +Rollback is to restore affected catalog constants to `LfsPath` and keep the LFS archives in place. + +## Open Questions + +- Exact upstream robot description source URLs and refs for Piper and A750 need confirmation during implementation. +- Whether GitPython should be a core dependency or an optional extra depends on whether migrated manipulator catalogs are imported in minimal DimOS installs. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/docs.md b/openspec/changes/migrate-robot-arms-to-git-assets/docs.md new file mode 100644 index 0000000000..5ebf2d4d7c --- /dev/null +++ b/openspec/changes/migrate-robot-arms-to-git-assets/docs.md @@ -0,0 +1,31 @@ +## User-Facing Docs + +- Update `docs/capabilities/manipulation/adding_a_custom_arm.md` so new custom arm guidance uses Robot Asset Manager declarations instead of presenting `LfsPath` bundles as the canonical path. +- Update manipulation capability docs that list model paths for xArm, Piper, and A750 so they describe robot asset declarations, artifact roles, ROS package roots, and cache behavior. +- Add or update a user-facing section under `docs/usage/` or `docs/capabilities/manipulation/` explaining: + - the Robot Asset Manager purpose, + - supported artifact roles (`urdf`, `mjcf`, `srdf`, `mesh_dir`, plus extra string roles), + - cache location and fresh-when-safe behavior, + - when commit refs should be used, + - why OpenArm remains on its current path for now. + +## Contributor Docs + +- Update contributor/development docs if implementation adds a new dependency or test workflow for robot assets. +- Document how to add a new typed robot asset declaration and how to choose package roots and artifact roles. +- Document test expectations for cache policy, dirty caches, update failures, and parser/planning compatibility. + +## Coding-Agent Docs + +- Update `AGENTS.md` or `docs/coding-agents/` only if the canonical workflow for adding robot model assets changes enough that coding agents need explicit instructions. +- Suggested coding-agent note: prefer Robot Asset Manager declarations for upstream robot description sources; use `LfsPath` only for assets that are intentionally vendored, locally modified, or not yet migrated. + +## Doc Validation + +- Run `uv run doclinks` after editing docs, if available in the environment. +- Run `md-babel-py run ` only for docs containing executable Python snippets. +- Run targeted tests referenced by docs if examples are made executable. + +## No Docs Needed + +Documentation changes are needed because this change replaces the documented LFS-based manipulator onboarding pattern and introduces user-visible cache/update behavior. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/proposal.md b/openspec/changes/migrate-robot-arms-to-git-assets/proposal.md new file mode 100644 index 0000000000..a622559259 --- /dev/null +++ b/openspec/changes/migrate-robot-arms-to-git-assets/proposal.md @@ -0,0 +1,39 @@ +## Why + +DimOS currently keeps several manipulation robot description bundles in Git LFS archives under `data/.lfs`. That makes routine development depend on copied binary/source asset bundles inside this repository, increases repository-specific asset maintenance, and makes it harder to track upstream robot description changes for supported arms. + +This change migrates the xArm, Piper, and A750 manipulation robot model assets to a Git-backed Robot Asset Manager. DimOS should resolve robot description sources from upstream repositories into a standard user cache while preserving the existing model-path and package-path behavior used by planning, control, parsing, simulation, and documentation. + +## What Changes + +- Add a Git-backed robot asset resolution capability for selected manipulation robot model files and package roots. +- Add typed Python robot asset declarations for xArm, Piper, and A750, with robot-model-first lookup and internal source checkout deduplication. +- Add a lazy Path-like adapter so existing `RobotConfig.model_path` and `package_paths` consumers can keep receiving filesystem paths without network access at import time. +- Migrate xArm, Piper, and A750 catalog model paths and package roots away from LFS-backed `LfsPath` declarations where suitable upstream robot description sources are available. +- Keep OpenArm on its current path for now because it has local DimOS modifications. +- Preserve Xacro, package URI, `$(find ...)`, MJCF/SRDF, and mesh directory compatibility through existing parser and Drake preparation layers. +- No **BREAKING** CLI, skill/MCP, or hardware-safety behavior changes are intended. + +## Affected DimOS Surfaces + +- Modules/streams: manipulation planning and control modules that consume `RobotConfig.model_path`, FK/IK model paths, package paths, Xacro inputs, MJCF files, SRDF files, and mesh directories. +- Blueprints/CLI: xArm, Piper, and A750 manipulator blueprints and teleop/control blueprint wiring that reference catalog constants; no CLI command behavior changes are intended. +- Skills/MCP: no direct skill or MCP tool behavior changes are intended. +- Hardware/simulation/replay: xArm, Piper, and A750 real/sim manipulation stacks may resolve model assets from the user cache instead of LFS bundles; OpenArm remains unchanged. +- Docs/generated registries: manipulation docs and tests that describe LFS-backed onboarding or hardcoded model bundle paths need updates; no generated blueprint registry changes are expected unless catalog exports change. + +## Capabilities + +### New Capabilities + +- `robot-asset-resolution`: Resolving robot model artifacts and ROS package roots from upstream robot description sources into local filesystem paths for DimOS consumers. + +### Modified Capabilities + +- None. + +## Impact + +Developers get easier robot asset maintenance and less need to copy upstream description bundles into DimOS. First use of a migrated arm may require network access to populate `~/.cache/dimos/robot_assets`; later uses can continue from the cached copy if upstream update checks fail. Clean cached repositories update when upstream changes are available, while dirty cached repositories are preserved with warnings. + +Compatibility risk is concentrated around upstream repository layout differences, package path resolution, Xacro expansion, and FK/IK model constants. The change needs unit tests for cache/update policy and asset path resolution, plus integration coverage for xArm, Piper, and A750 catalog paths through existing model parsing/planning entry points. Documentation should explain the new asset declarations, cache behavior, supported artifact roles, and the temporary split where OpenArm remains LFS-backed. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md b/openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md new file mode 100644 index 0000000000..3fd153ecda --- /dev/null +++ b/openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md @@ -0,0 +1,93 @@ +## ADDED Requirements + +### Requirement: Resolve robot model artifacts by role + +DimOS SHALL resolve declared robot model artifacts for supported manipulation robot models by robot model name and artifact role. + +#### Scenario: Resolve a migrated arm URDF +- **GIVEN** a migrated robot model such as xArm, Piper, or A750 has a declared `urdf` artifact role +- **WHEN** a DimOS catalog or model consumer requests that artifact +- **THEN** DimOS returns a local filesystem path to the declared URDF or Xacro file +- **AND** existing model parsing consumers can use the returned path as a normal path-like model input. + +#### Scenario: Resolve additional flat artifact roles +- **GIVEN** a migrated robot model declares artifact roles such as `mjcf`, `srdf`, `mesh_dir`, or an extra string role such as `urdf_ik` +- **WHEN** a consumer requests one of those declared roles +- **THEN** DimOS returns the local filesystem path for that role +- **AND** DimOS reports an explicit error when the requested role is not declared for that robot model. + +### Requirement: Resolve ROS package roots for model consumers + +DimOS SHALL provide ROS package root mappings for migrated robot models when their model files require package-based resource resolution. + +#### Scenario: Resolve package URI resources +- **GIVEN** a migrated robot model declares a ROS package root for a package used by `package://...` URIs +- **WHEN** a parser, planner, or Drake preparation layer receives the model path and package roots +- **THEN** the consumer can resolve package-based mesh and resource references using the declared package root +- **AND** DimOS preserves compatibility with existing `dict[str, Path]` package root consumers. + +#### Scenario: Resolve Xacro find expressions +- **GIVEN** a migrated robot model declares a ROS package root for a package used by `$(find package_name)` in Xacro +- **WHEN** existing Xacro processing expands the model file +- **THEN** the package name resolves to the declared local package root +- **AND** Xacro processing remains the responsibility of the existing parser or Drake preparation layer. + +### Requirement: Populate and reuse a standard robot asset cache + +DimOS SHALL store fetched robot description sources in a standard user cache and reuse cached sources across runs. + +#### Scenario: Cache is missing +- **GIVEN** a requested migrated robot model has no cached source checkout +- **WHEN** DimOS resolves one of its artifacts +- **THEN** DimOS fetches the declared robot description source into the standard robot asset cache +- **AND** DimOS fails with a clear error if the source cannot be fetched and no cached checkout exists. + +#### Scenario: Cache is present and update succeeds +- **GIVEN** a requested migrated robot model has a clean cached source checkout +- **WHEN** DimOS resolves one of its artifacts and the upstream source has changed for the declared ref +- **THEN** DimOS updates the cached checkout before returning the artifact path +- **AND** the returned path points into the updated cached checkout. + +#### Scenario: Cache is present and update fails +- **GIVEN** a requested migrated robot model has a cached source checkout +- **WHEN** DimOS cannot check for or apply an upstream update +- **THEN** DimOS warns about the update failure +- **AND** DimOS continues using the cached checkout. + +#### Scenario: Cache has local changes +- **GIVEN** a requested migrated robot model has a cached source checkout with local changes +- **WHEN** DimOS resolves one of its artifacts +- **THEN** DimOS warns that the cache has local changes and skips upstream update +- **AND** DimOS returns paths from the dirty cached checkout without overwriting local edits. + +### Requirement: Preserve catalog path compatibility + +DimOS SHALL expose migrated robot assets through catalog declarations that remain compatible with existing path-based robot model consumers. + +#### Scenario: Catalog import remains lightweight +- **GIVEN** a module imports xArm, Piper, or A750 catalog constants +- **WHEN** the import completes +- **THEN** DimOS does not fetch robot description sources during import +- **AND** any network or cache resolution work is deferred until a concrete filesystem path is needed. + +#### Scenario: Existing RobotConfig consumers receive compatible paths +- **GIVEN** a migrated catalog creates a DimOS Robot Model Config with a model path and package roots +- **WHEN** an existing planner, parser, simulation, or visualization consumer reads that config +- **THEN** the model path behaves as a path-like filesystem value +- **AND** package roots remain compatible with existing `dict[str, Path]` expectations. + +### Requirement: Support flexible source refs + +DimOS SHALL support branch, tag, and commit refs in robot asset declarations for migrated robot models. + +#### Scenario: Declaration uses branch or tag ref +- **GIVEN** a migrated robot model declaration uses an upstream branch or tag ref +- **WHEN** DimOS resolves an artifact for that model +- **THEN** DimOS fetches and checks out the declared ref according to the cache freshness policy +- **AND** developers can use upstream-moving refs when ease of use and freshness are preferred. + +#### Scenario: Declaration uses commit ref +- **GIVEN** a migrated robot model declaration uses a commit ref +- **WHEN** DimOS resolves an artifact for that model +- **THEN** DimOS checks out that commit in the cache +- **AND** releases, CI, or fragile assets can use pinned refs when stronger reproducibility is required. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/tasks.md b/openspec/changes/migrate-robot-arms-to-git-assets/tasks.md new file mode 100644 index 0000000000..eb7b22c397 --- /dev/null +++ b/openspec/changes/migrate-robot-arms-to-git-assets/tasks.md @@ -0,0 +1,36 @@ +## 1. Implementation + +- [x] 1.1 Add runtime dependencies for Git-backed asset resolution (`GitPython` and `filelock`) in project dependency configuration. +- [x] 1.2 Implement a generic Git asset cache with clone-on-miss, clean-cache update, update-failure fallback, dirty-cache skip, and per-source file locking. +- [x] 1.3 Add unit tests for Git asset cache behavior, including missing cache failure, successful clean update, update failure with cached fallback, and dirty cache preservation. +- [x] 1.4 Implement typed robot asset declarations with robot-model-first lookup, flat artifact role strings, ROS package roots, optional model-level Xacro args, and source checkout deduplication by `(repo_url, ref)`. +- [x] 1.5 Implement `RobotAssetManager` resolution for artifact paths and package roots, including explicit errors for unknown robot models or undeclared artifact roles. +- [x] 1.6 Implement `RobotAssetPath` as a lazy Path-like catalog adapter that avoids network/filesystem resolution at import time and resolves only when a concrete path is needed. +- [x] 1.7 Add unit tests for robot asset declaration lookup, package root resolution, lazy import behavior, and common path-like operations used by existing consumers. +- [x] 1.8 Identify and verify upstream robot description source URLs, refs, artifact paths, and ROS package roots for xArm, Piper, and A750. +- [x] 1.9 Add xArm, Piper, and A750 robot asset declarations for `urdf`, any required FK/IK URDF role such as `urdf_ik`, `mjcf` where applicable, `srdf` where applicable, and `mesh_dir` where applicable. +- [x] 1.10 Migrate xArm catalog constants and package roots from `LfsPath` to robot asset declarations and `RobotAssetPath`. +- [x] 1.11 Migrate Piper catalog constants and package roots from `LfsPath` to robot asset declarations and `RobotAssetPath`. +- [x] 1.12 Migrate A750 catalog constants and package roots from `LfsPath` to robot asset declarations and `RobotAssetPath`. +- [x] 1.13 Keep OpenArm catalog behavior unchanged and add comments/tests where useful to make the intentional non-migration clear. +- [x] 1.14 Update tests or fixtures that directly reference `get_data("xarm_description")`, Piper LFS paths, or A750 LFS paths so they validate the new asset resolution path instead. +- [x] 1.15 If catalog exports, blueprint names, or generated registry inputs change, regenerate and verify the blueprint registry with `pytest dimos/robot/test_all_blueprints_generation.py`. + +## 2. Documentation + +- [x] 2.1 Update `docs/capabilities/manipulation/adding_a_custom_arm.md` to describe Robot Asset Manager declarations as the preferred upstream robot description source workflow. +- [x] 2.2 Update xArm, Piper, and A750 manipulation docs to describe resolved artifact roles, ROS package roots, cache behavior, and any changed model path examples. +- [x] 2.3 Add or update a user-facing Robot Asset Manager documentation section covering purpose, cache location, fresh-when-safe behavior, supported artifact roles, and branch/tag/commit ref guidance. +- [x] 2.4 Update contributor docs if new dependency, cache testing, or robot asset declaration workflow details need contributor guidance. +- [x] 2.5 Update `AGENTS.md` or `docs/coding-agents/` if the implementation changes the recommended coding-agent workflow for adding robot model assets. + +## 3. Verification + +- [x] 3.1 Run `openspec validate migrate-robot-arms-to-git-assets`. +- [x] 3.2 Run focused unit tests for the Git asset cache and robot asset manager. +- [x] 3.3 Run focused catalog/model parsing tests for xArm, Piper, and A750 migrated paths. +- [x] 3.4 Run existing manipulation planning/control tests affected by catalog model path changes, including the current `dimos/manipulation/test_manipulation_module.py` target if still applicable. +- [x] 3.5 Run `uv run pytest dimos/robot/test_all_blueprints_generation.py` if blueprint registry inputs or generated registry output may have changed. +- [x] 3.6 Run `uv run doclinks` after documentation updates. +- [x] 3.7 Run `md-babel-py run ` for any changed documentation file that contains executable Python snippets. +- [x] 3.8 Manually QA at least one migrated xArm path and one non-xArm migrated path through parser/planning or simulation/replay before any real hardware use. diff --git a/pyproject.toml b/pyproject.toml index 6806733e06..d3cab684fd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -107,6 +107,8 @@ dependencies = [ "annotation-protocol>=1.4.0", "lazy_loader", "plum-dispatch==2.5.7", + "GitPython>=3.1.45", + "filelock>=3.20.0", # Logging "structlog>=25.5.0,<26", "colorlog==6.9.0", diff --git a/uv.lock b/uv.lock index 0752247aa5..9f8d54f4f8 100644 --- a/uv.lock +++ b/uv.lock @@ -1858,6 +1858,8 @@ dependencies = [ { name = "cryptography" }, { name = "dimos-lcm" }, { name = "dimos-viewer" }, + { name = "filelock" }, + { name = "gitpython" }, { name = "lazy-loader" }, { name = "llvmlite" }, { name = "lz4" }, @@ -2360,8 +2362,10 @@ requires-dist = [ { name = "fastapi", marker = "extra == 'web'", specifier = ">=0.115.6" }, { name = "faster-whisper", marker = "extra == 'agents'", specifier = ">=1.0.0" }, { name = "ffmpeg-python", marker = "extra == 'web'" }, + { name = "filelock", specifier = ">=3.20.0" }, { name = "filterpy", marker = "extra == 'perception'", specifier = ">=1.4.5" }, { name = "gdown", marker = "extra == 'misc'", specifier = ">=5.2.2" }, + { name = "gitpython", specifier = ">=3.1.45" }, { name = "googlemaps", marker = "extra == 'misc'", specifier = ">=4.10.0" }, { name = "gtsam-extended", marker = "extra == 'mapping'", specifier = ">=4.3a1.post1" }, { name = "hydra-core", marker = "extra == 'perception'", specifier = ">=1.3.0" }, @@ -3383,6 +3387,30 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/91/fd/a382bb6684b1fdbe5cd19aa980a04a67f6c91efd0e1e627f93614fe2d24e/gdown-6.0.0-py3-none-any.whl", hash = "sha256:c82d39a6b09ed7778012515c2fa4ab4dc36d7789300cd0b16b87d3a3e4a09955", size = 18243, upload-time = "2026-04-12T06:37:38.209Z" }, ] +[[package]] +name = "gitdb" +version = "4.0.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "smmap" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/72/94/63b0fc47eb32792c7ba1fe1b694daec9a63620db1e313033d18140c2320a/gitdb-4.0.12.tar.gz", hash = "sha256:5ef71f855d191a3326fcfbc0d5da835f26b13fbcba60c32c21091c349ffdb571", size = 394684, upload-time = "2025-01-02T07:20:46.413Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" }, +] + +[[package]] +name = "gitpython" +version = "3.1.50" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "gitdb" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/33/f6/354ae6491228b5eb40e10d89c4d13c651fe1cf7556e35ebdded50cff57ce/gitpython-3.1.50.tar.gz", hash = "sha256:80da2d12504d52e1f998772dc5baf6e553f8d2fcfe1fcc226c9d9a2ee3372dcc", size = 219798, upload-time = "2026-05-06T04:01:26.571Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/20/7a/1c6e3562dfd8950adbb11ffbc65d21e7c89d01a6e4f137fa981056de25c5/gitpython-3.1.50-py3-none-any.whl", hash = "sha256:d352abe2908d07355014abdd21ddf798c2a961469239afec4962e9da884858f9", size = 212507, upload-time = "2026-05-06T04:01:23.799Z" }, +] + [[package]] name = "glfw" version = "2.10.0" @@ -9920,6 +9948,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] +[[package]] +name = "smmap" +version = "5.0.3" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" }, +] + [[package]] name = "sniffio" version = "1.3.1" From 711c8393c6d53d654a9c8d9310063dfda20d9ab2 Mon Sep 17 00:00:00 2001 From: cc Date: Mon, 15 Jun 2026 22:26:27 -0700 Subject: [PATCH 04/15] refactor: isolate robot asset package --- .../manipulation/planning/utils/mesh_utils.py | 91 +----- .../planning/utils/test_mesh_utils.py | 52 +++ .../planning/world/drake_world.py | 5 +- .../manipulation/test_manipulation_module.py | 2 +- dimos/robot/asset_manager.py | 295 ++---------------- dimos/robot/asset_processing.py | 31 ++ dimos/robot/assets/__init__.py | 65 ++++ dimos/robot/assets/declarations.py | 79 +++++ dimos/robot/assets/git_cache.py | 171 ++++++++++ dimos/robot/assets/manager.py | 294 +++++++++++++++++ dimos/robot/assets/processing.py | 152 +++++++++ dimos/robot/assets/test_processing.py | 62 ++++ dimos/robot/catalog/a750.py | 2 +- dimos/robot/catalog/piper.py | 2 +- dimos/robot/catalog/ufactory.py | 14 +- dimos/robot/robot_asset_declarations.py | 76 +---- dimos/robot/test_asset_manager.py | 37 ++- dimos/utils/git_asset_cache.py | 176 ++--------- dimos/utils/test_git_asset_cache.py | 2 +- 19 files changed, 1030 insertions(+), 578 deletions(-) create mode 100644 dimos/manipulation/planning/utils/test_mesh_utils.py create mode 100644 dimos/robot/asset_processing.py create mode 100644 dimos/robot/assets/__init__.py create mode 100644 dimos/robot/assets/declarations.py create mode 100644 dimos/robot/assets/git_cache.py create mode 100644 dimos/robot/assets/manager.py create mode 100644 dimos/robot/assets/processing.py create mode 100644 dimos/robot/assets/test_processing.py diff --git a/dimos/manipulation/planning/utils/mesh_utils.py b/dimos/manipulation/planning/utils/mesh_utils.py index 7180c4f896..3270e13476 100644 --- a/dimos/manipulation/planning/utils/mesh_utils.py +++ b/dimos/manipulation/planning/utils/mesh_utils.py @@ -32,13 +32,12 @@ from __future__ import annotations import hashlib -import os from pathlib import Path import re import shutil -import tempfile from typing import TYPE_CHECKING +from dimos.robot.assets.processing import DERIVED_ASSET_CACHE_ROOT, render_urdf from dimos.utils.logging_config import setup_logger if TYPE_CHECKING: @@ -47,8 +46,8 @@ logger = setup_logger() -# Cache directory for processed URDFs -_CACHE_DIR = Path(tempfile.gettempdir()) / "dimos_urdf_cache" +# Cache directory for Drake-specific URDFs derived from rendered robot assets. +_CACHE_DIR = DERIVED_ASSET_CACHE_ROOT / "drake_urdfs" def prepare_urdf_for_drake( @@ -73,33 +72,31 @@ def prepare_urdf_for_drake( Returns: Path to the prepared URDF file (may be cached) """ - urdf_path = Path(os.fspath(urdf_path)) package_paths = package_paths or {} xacro_args = xacro_args or {} + rendered_urdf = render_urdf( + urdf_path, + package_paths, + xacro_args, + package_uri_mode="absolute", + ) - # Generate cache key - cache_key = _generate_cache_key(urdf_path, package_paths, xacro_args, convert_meshes) - cache_path = _CACHE_DIR / cache_key / urdf_path.stem + # Generate cache key for Drake-specific processing. + cache_key = _generate_cache_key(rendered_urdf, convert_meshes) + cache_path = _CACHE_DIR / cache_key / rendered_urdf.stem cache_path.mkdir(parents=True, exist_ok=True) - cached_urdf = cache_path / f"{urdf_path.stem}.urdf" + cached_urdf = cache_path / f"{rendered_urdf.stem}.urdf" # Check cache if cached_urdf.exists(): logger.debug(f"Using cached URDF: {cached_urdf}") return str(cached_urdf) - # Process xacro if needed - if urdf_path.suffix in (".xacro", ".urdf.xacro"): - urdf_content = _process_xacro(urdf_path, package_paths, xacro_args) - else: - urdf_content = urdf_path.read_text() + urdf_content = rendered_urdf.read_text() # Strip transmission blocks (Drake doesn't need them, and they can cause issues) urdf_content = _strip_transmission_blocks(urdf_content) - # Resolve package:// URIs - urdf_content = _resolve_package_uris(urdf_content, package_paths, cache_path) - # Convert meshes if requested if convert_meshes: urdf_content = _convert_meshes(urdf_content, cache_path) @@ -113,11 +110,9 @@ def prepare_urdf_for_drake( def _generate_cache_key( urdf_path: Path, - package_paths: dict[str, Path], - xacro_args: dict[str, str], convert_meshes: bool, ) -> str: - """Generate a cache key for the URDF configuration. + """Generate a cache key for Drake-specific URDF processing. Includes a version number to invalidate cache when processing logic changes. """ @@ -126,33 +121,11 @@ def _generate_cache_key( # Version number to invalidate cache when processing logic changes # Increment this when adding new processing steps (e.g., stripping transmission blocks) - processing_version = "v2" - - resolved_package_paths = { - package_name: Path(os.fspath(package_path)).resolve() - for package_name, package_path in package_paths.items() - } - key_data = f"{processing_version}:{urdf_path}:{mtime}:{sorted(resolved_package_paths.items())}:{sorted(xacro_args.items())}:{convert_meshes}" + processing_version = "drake-urdf-v1" + key_data = f"{processing_version}:{urdf_path}:{mtime}:{convert_meshes}" return hashlib.md5(key_data.encode()).hexdigest()[:16] -def _process_xacro( - xacro_path: Path, - package_paths: dict[str, Path], - xacro_args: dict[str, str], -) -> str: - """Process xacro file to URDF.""" - try: - from dimos.utils.ament_prefix import process_xacro - except ImportError: - raise ImportError( - "xacro is required for processing .xacro files. " - "Install the manipulation extra: pip install dimos[manipulation]" - ) - - return process_xacro(xacro_path, package_paths, xacro_args) - - def _strip_transmission_blocks(urdf_content: str) -> str: """Remove transmission blocks from URDF content. @@ -180,36 +153,6 @@ def _strip_transmission_blocks(urdf_content: str) -> str: return result -def _resolve_package_uris( - urdf_content: str, - package_paths: dict[str, Path], - output_dir: Path, -) -> str: - """Resolve package:// URIs to filesystem paths.""" - # Pattern for package:// URIs (handles both single and double quotes) - # Note: Use triple quotes so \s is correctly interpreted as whitespace, not literal 's' - pattern = r"""package://([^/]+)/(.+?)(["'<>\s])""" - - def replace_uri(match: re.Match[str]) -> str: - pkg_name = match.group(1) - rel_path = match.group(2) - suffix = match.group(3) - - if pkg_name in package_paths: - # Ensure absolute path for proper resolution - pkg_path = Path(os.fspath(package_paths[pkg_name])).resolve() - full_path = pkg_path / rel_path - if full_path.exists(): - return f"{full_path}{suffix}" - else: - logger.warning(f"File not found: {full_path}") - - # Return original if not found - return match.group(0) - - return re.sub(pattern, replace_uri, urdf_content) - - def _convert_meshes(urdf_content: str, output_dir: Path) -> str: """Convert DAE/STL meshes to OBJ format for Drake compatibility.""" try: diff --git a/dimos/manipulation/planning/utils/test_mesh_utils.py b/dimos/manipulation/planning/utils/test_mesh_utils.py new file mode 100644 index 0000000000..285b7a3115 --- /dev/null +++ b/dimos/manipulation/planning/utils/test_mesh_utils.py @@ -0,0 +1,52 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +from dimos.manipulation.planning.utils import mesh_utils +from dimos.robot.assets import processing + + +def test_prepare_urdf_for_drake_uses_rendered_urdf_and_keeps_drake_cleanup( + tmp_path: Path, + monkeypatch, +) -> None: + monkeypatch.setattr(processing, "_RENDERED_URDF_CACHE_ROOT", tmp_path / "rendered") + monkeypatch.setattr(mesh_utils, "_CACHE_DIR", tmp_path / "drake") + package_root = tmp_path / "pkg" + mesh = package_root / "meshes" / "link.stl" + mesh.parent.mkdir(parents=True) + mesh.write_text("solid link\nendsolid link\n") + urdf = tmp_path / "robot.urdf" + urdf.write_text( + "" + "" + "" + "" + "bad" + "" + ) + + prepared = Path( + mesh_utils.prepare_urdf_for_drake( + urdf, + {"pkg": package_root}, + ) + ) + + prepared_text = prepared.read_text() + assert prepared.is_relative_to(tmp_path / "drake") + assert "package://" not in prepared_text + assert str(mesh) in prepared_text + assert " WorldRobotID: def _load_model(self, config: RobotModelConfig) -> Any: """Load robot model (URDF/xacro/MJCF) and return model instance.""" - original_path = Path(os.fspath(config.model_path)).resolve() + original_path = config.model_path.resolve() if not original_path.exists(): raise FileNotFoundError(f"Robot model not found: {original_path}") @@ -257,7 +256,7 @@ def _load_model(self, config: RobotModelConfig) -> Any: # Register package paths (not applicable to MJCF) if config.package_paths: for pkg_name, pkg_path in config.package_paths.items(): - self._parser.package_map().Add(pkg_name, Path(os.fspath(pkg_path))) + self._parser.package_map().Add(pkg_name, pkg_path) else: self._parser.package_map().Add( f"{config.name}_description", prepared_path_obj.parent diff --git a/dimos/manipulation/test_manipulation_module.py b/dimos/manipulation/test_manipulation_module.py index f6999ade24..d2a886d63e 100644 --- a/dimos/manipulation/test_manipulation_module.py +++ b/dimos/manipulation/test_manipulation_module.py @@ -36,7 +36,7 @@ from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 from dimos.msgs.sensor_msgs.JointState import JointState -from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets import RobotAssetPath, robot_asset_package_paths pytestmark = pytest.mark.self_hosted diff --git a/dimos/robot/asset_manager.py b/dimos/robot/asset_manager.py index 7f057dacc3..d3f70176ee 100644 --- a/dimos/robot/asset_manager.py +++ b/dimos/robot/asset_manager.py @@ -12,271 +12,30 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Robot model asset declarations and lazy path adapters.""" - -from __future__ import annotations - -from collections.abc import Mapping -from dataclasses import dataclass, field -from enum import Enum -from pathlib import Path - -from dimos.utils.git_asset_cache import GitAssetCache - - -class RobotAssetError(RuntimeError): - """Raised when a robot asset declaration cannot satisfy a request.""" - - -class ArtifactRole(str, Enum): - """Common robot asset artifact roles. - - Strings are canonical internally; this enum is a convenience for common roles. - """ - - URDF = "urdf" - MJCF = "mjcf" - SRDF = "srdf" - MESH_DIR = "mesh_dir" - - -@dataclass(frozen=True) -class RobotAssetDeclaration: - """Typed declaration for one robot model's assets.""" - - model: str - repo_url: str - ref: str - artifacts: Mapping[str, str] - package_roots: Mapping[str, str] = field(default_factory=dict) - xacro_args: Mapping[str, str] = field(default_factory=dict) - source_name: str | None = None - license: str | None = None - - -class RobotAssetManager: - """Resolve robot model artifacts and package roots from declarations.""" - - def __init__( - self, - declarations: Mapping[str, RobotAssetDeclaration] | None = None, - git_cache: GitAssetCache | None = None, - ) -> None: - self._declarations = dict(declarations or {}) - self._git_cache = git_cache or GitAssetCache() - - def get_declaration(self, model: str) -> RobotAssetDeclaration: - try: - return self._declarations[model] - except KeyError as exc: - available = ", ".join(sorted(self._declarations)) or "none" - raise RobotAssetError( - f"Unknown robot asset model {model!r}. Available models: {available}." - ) from exc - - def resolve_artifact(self, model: str, role: str | ArtifactRole) -> Path: - declaration = self.get_declaration(model) - role_key = _role_key(role) - try: - relative_path = declaration.artifacts[role_key] - except KeyError as exc: - available = ", ".join(sorted(declaration.artifacts)) or "none" - raise RobotAssetError( - f"Robot asset model {model!r} does not declare artifact role {role_key!r}. " - f"Available roles: {available}." - ) from exc - - path = self._checkout(declaration) / relative_path - if not path.exists(): - raise RobotAssetError( - f"Declared artifact {role_key!r} for robot asset model {model!r} does not exist: {path}" - ) - return path - - def resolve_package_root(self, model: str, package_name: str) -> Path: - declaration = self.get_declaration(model) - try: - relative_path = declaration.package_roots[package_name] - except KeyError as exc: - available = ", ".join(sorted(declaration.package_roots)) or "none" - raise RobotAssetError( - f"Robot asset model {model!r} does not declare ROS package root " - f"{package_name!r}. Available package roots: {available}." - ) from exc - - path = self._checkout(declaration) / relative_path - if not path.exists(): - raise RobotAssetError( - f"Declared package root {package_name!r} for robot asset model {model!r} " - f"does not exist: {path}" - ) - return path - - def package_roots(self, model: str) -> dict[str, Path]: - declaration = self.get_declaration(model) - return { - package_name: RobotAssetPackagePath(model, package_name, manager=self) - for package_name in declaration.package_roots - } - - def xacro_args(self, model: str) -> dict[str, str]: - return dict(self.get_declaration(model).xacro_args) - - def _checkout(self, declaration: RobotAssetDeclaration) -> Path: - return self._git_cache.resolve(declaration.repo_url, declaration.ref).path - - -class RobotAssetPath(type(Path())): # type: ignore[misc] - """Lazy Path-like adapter for a declared robot model artifact.""" - - def __new__( - cls, - model: str, - role: str | ArtifactRole, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> RobotAssetPath: - instance: RobotAssetPath = super().__new__(cls, ".") - object.__setattr__(instance, "_robot_asset_model", model) - object.__setattr__(instance, "_robot_asset_role", _role_key(role)) - object.__setattr__( - instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) - ) - object.__setattr__( - instance, "_robot_asset_manager", manager or default_robot_asset_manager() - ) - object.__setattr__(instance, "_robot_asset_resolved_cache", None) - return instance - - def __init__( - self, - model: str, - role: str | ArtifactRole, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> None: - # Path subclasses receive the same constructor arguments in __init__ after - # __new__; consume them so pathlib does not see DimOS-specific kwargs. - del model, role, relative_parts, manager - - def _resolve(self) -> Path: - cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") - if cache is None: - manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") - model = object.__getattribute__(self, "_robot_asset_model") - role = object.__getattribute__(self, "_robot_asset_role") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - cache = manager.resolve_artifact(model, role).joinpath(*relative_parts) - object.__setattr__(self, "_robot_asset_resolved_cache", cache) - return cache - - def __getattribute__(self, name: str) -> object: - try: - object.__getattribute__(self, "_robot_asset_model") - except AttributeError: - return object.__getattribute__(self, name) - - if name.startswith("_robot_asset_") or name in {"_resolve"}: - return object.__getattribute__(self, name) - - return getattr(object.__getattribute__(self, "_resolve")(), name) - - def __str__(self) -> str: - return str(self._resolve()) - - def __fspath__(self) -> str: - return str(self._resolve()) - - def __truediv__(self, other: object) -> RobotAssetPath: - model = object.__getattribute__(self, "_robot_asset_model") - role = object.__getattribute__(self, "_robot_asset_role") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - manager = object.__getattribute__(self, "_robot_asset_manager") - return RobotAssetPath(model, role, *relative_parts, other, manager=manager) - - -class RobotAssetPackagePath(type(Path())): # type: ignore[misc] - """Lazy Path-like adapter for a declared ROS package root.""" - - def __new__( - cls, - model: str, - package_name: str, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> RobotAssetPackagePath: - instance: RobotAssetPackagePath = super().__new__(cls, ".") - object.__setattr__(instance, "_robot_asset_model", model) - object.__setattr__(instance, "_robot_asset_package_name", package_name) - object.__setattr__( - instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) - ) - object.__setattr__( - instance, "_robot_asset_manager", manager or default_robot_asset_manager() - ) - object.__setattr__(instance, "_robot_asset_resolved_cache", None) - return instance - - def __init__( - self, - model: str, - package_name: str, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> None: - del model, package_name, relative_parts, manager - - def _resolve(self) -> Path: - cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") - if cache is None: - manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") - model = object.__getattribute__(self, "_robot_asset_model") - package_name = object.__getattribute__(self, "_robot_asset_package_name") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - cache = manager.resolve_package_root(model, package_name).joinpath(*relative_parts) - object.__setattr__(self, "_robot_asset_resolved_cache", cache) - return cache - - def __getattribute__(self, name: str) -> object: - try: - object.__getattribute__(self, "_robot_asset_model") - except AttributeError: - return object.__getattribute__(self, name) - - if name.startswith("_robot_asset_") or name in {"_resolve"}: - return object.__getattribute__(self, name) - - return getattr(object.__getattribute__(self, "_resolve")(), name) - - def __str__(self) -> str: - return str(self._resolve()) - - def __fspath__(self) -> str: - return str(self._resolve()) - - def __truediv__(self, other: object) -> RobotAssetPackagePath: - model = object.__getattribute__(self, "_robot_asset_model") - package_name = object.__getattribute__(self, "_robot_asset_package_name") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - manager = object.__getattribute__(self, "_robot_asset_manager") - return RobotAssetPackagePath(model, package_name, *relative_parts, other, manager=manager) - - -_DEFAULT_MANAGER: RobotAssetManager | None = None - - -def default_robot_asset_manager() -> RobotAssetManager: - global _DEFAULT_MANAGER - if _DEFAULT_MANAGER is None: - from dimos.robot.robot_asset_declarations import ROBOT_ASSETS - - _DEFAULT_MANAGER = RobotAssetManager(ROBOT_ASSETS) - return _DEFAULT_MANAGER - - -def robot_asset_package_paths(model: str) -> dict[str, Path]: - return default_robot_asset_manager().package_roots(model) - - -def _role_key(role: str | ArtifactRole) -> str: - return role.value if isinstance(role, ArtifactRole) else str(role) +"""Compatibility exports for :mod:`dimos.robot.assets.manager`.""" + +from dimos.robot.assets.manager import ( + ArtifactRole, + RobotAssetDeclaration, + RobotAssetError, + RobotAssetManager, + RobotAssetPackagePath, + RobotAssetPath, + default_robot_asset_manager, + robot_asset_package_paths, + robot_asset_xacro_args, + set_default_robot_asset_manager, +) + +__all__ = [ + "ArtifactRole", + "RobotAssetDeclaration", + "RobotAssetError", + "RobotAssetManager", + "RobotAssetPackagePath", + "RobotAssetPath", + "default_robot_asset_manager", + "robot_asset_package_paths", + "robot_asset_xacro_args", + "set_default_robot_asset_manager", +] diff --git a/dimos/robot/asset_processing.py b/dimos/robot/asset_processing.py new file mode 100644 index 0000000000..3f1bedfab2 --- /dev/null +++ b/dimos/robot/asset_processing.py @@ -0,0 +1,31 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Compatibility exports for :mod:`dimos.robot.assets.processing`.""" + +from dimos.robot.assets.processing import ( + DERIVED_ASSET_CACHE_ROOT, + PackageUriMode, + normalize_package_paths, + render_urdf, + resolve_package_uris, +) + +__all__ = [ + "DERIVED_ASSET_CACHE_ROOT", + "PackageUriMode", + "normalize_package_paths", + "render_urdf", + "resolve_package_uris", +] diff --git a/dimos/robot/assets/__init__.py b/dimos/robot/assets/__init__.py new file mode 100644 index 0000000000..e546727a31 --- /dev/null +++ b/dimos/robot/assets/__init__.py @@ -0,0 +1,65 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Git-backed robot asset resolution and universal asset processing.""" + +from dimos.robot.assets.git_cache import ( + DEFAULT_GIT_ASSET_CACHE_ROOT, + DEFAULT_ROBOT_ASSET_CACHE_ROOT, + GitAssetCache, + GitAssetCacheError, + GitAssetCacheWarning, + GitAssetCheckout, +) +from dimos.robot.assets.manager import ( + ArtifactRole, + RobotAssetDeclaration, + RobotAssetError, + RobotAssetManager, + RobotAssetPackagePath, + RobotAssetPath, + default_robot_asset_manager, + robot_asset_package_paths, + robot_asset_xacro_args, + set_default_robot_asset_manager, +) +from dimos.robot.assets.processing import ( + DERIVED_ASSET_CACHE_ROOT, + PackageUriMode, + render_urdf, + resolve_package_uris, +) + +__all__ = [ + "DEFAULT_GIT_ASSET_CACHE_ROOT", + "DEFAULT_ROBOT_ASSET_CACHE_ROOT", + "DERIVED_ASSET_CACHE_ROOT", + "ArtifactRole", + "GitAssetCache", + "GitAssetCacheError", + "GitAssetCacheWarning", + "GitAssetCheckout", + "PackageUriMode", + "RobotAssetDeclaration", + "RobotAssetError", + "RobotAssetManager", + "RobotAssetPackagePath", + "RobotAssetPath", + "default_robot_asset_manager", + "render_urdf", + "resolve_package_uris", + "robot_asset_package_paths", + "robot_asset_xacro_args", + "set_default_robot_asset_manager", +] diff --git a/dimos/robot/assets/declarations.py b/dimos/robot/assets/declarations.py new file mode 100644 index 0000000000..a41fbe5ae4 --- /dev/null +++ b/dimos/robot/assets/declarations.py @@ -0,0 +1,79 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Robot asset declarations resolved by :mod:`dimos.robot.assets`.""" + +from __future__ import annotations + +from dimos.robot.assets.manager import RobotAssetDeclaration + +XARM_ROS2_REPO = "https://github.com/xArm-Developer/xarm_ros2" +PIPER_DESCRIPTION_REPO = "https://github.com/agilexrobotics/agx_arm_urdf" +A750_DESCRIPTION_REPO = "https://github.com/adob/a750_description" + + +ROBOT_ASSETS: dict[str, RobotAssetDeclaration] = { + "xarm6": RobotAssetDeclaration( + model="xarm6", + repo_url=XARM_ROS2_REPO, + ref="humble", + artifacts={ + "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", + "mesh_dir": "xarm_description/meshes", + }, + package_roots={"xarm_description": "xarm_description"}, + xacro_args={"dof": "6", "limited": "true"}, + source_name="xarm_ros2", + ), + "xarm7": RobotAssetDeclaration( + model="xarm7", + repo_url=XARM_ROS2_REPO, + ref="humble", + artifacts={ + "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", + "mesh_dir": "xarm_description/meshes", + }, + package_roots={"xarm_description": "xarm_description"}, + xacro_args={"dof": "7", "limited": "true"}, + source_name="xarm_ros2", + ), + "piper": RobotAssetDeclaration( + model="piper", + repo_url=PIPER_DESCRIPTION_REPO, + ref="main", + artifacts={ + "urdf": "piper/urdf/piper_with_gripper_description.xacro", + "urdf_ik": "piper/urdf/piper_description.urdf", + "mesh_dir": "piper/meshes", + }, + # Upstream URDFs reference package://agx_arm_description/agx_arm_urdf/... + # and expect the checkout directory to be named agx_arm_urdf inside the + # package root. GitAssetCache preserves that checkout directory name. + package_roots={"agx_arm_description": ".."}, + source_name="agx_arm_urdf", + license="MIT", + ), + "a750": RobotAssetDeclaration( + model="a750", + repo_url=A750_DESCRIPTION_REPO, + ref="master", + artifacts={ + "urdf": "urdf/a750_rev1.urdf", + "mesh_dir": "meshes/a750_rev1", + }, + package_roots={"a750_description": "."}, + source_name="a750_description", + license="MIT", + ), +} diff --git a/dimos/robot/assets/git_cache.py b/dimos/robot/assets/git_cache.py new file mode 100644 index 0000000000..4c6b9f7149 --- /dev/null +++ b/dimos/robot/assets/git_cache.py @@ -0,0 +1,171 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Git-backed cache for source repositories and other assets.""" + +from __future__ import annotations + +from dataclasses import dataclass +from hashlib import sha256 +from pathlib import Path +import shutil +import tempfile +from urllib.parse import urlparse +import warnings + +from filelock import FileLock +from git import GitCommandError, InvalidGitRepositoryError, NoSuchPathError, Repo + +DEFAULT_GIT_ASSET_CACHE_ROOT = Path.home() / ".cache" / "dimos" / "robot_assets" +DEFAULT_ROBOT_ASSET_CACHE_ROOT = DEFAULT_GIT_ASSET_CACHE_ROOT + + +class GitAssetCacheError(RuntimeError): + """Raised when an asset source cannot be resolved.""" + + +class GitAssetCacheWarning(RuntimeWarning): + """Warning emitted when a cached checkout is usable but not fresh.""" + + +@dataclass(frozen=True) +class GitAssetCheckout: + """Resolved local checkout information.""" + + path: Path + repo_url: str + ref: str + updated: bool = False + used_cached_fallback: bool = False + skipped_dirty_update: bool = False + + +class GitAssetCache: + """Resolve `(repo_url, ref)` pairs into fresh-when-safe cached checkouts. + + Policy: + - clone when the cache is missing; + - for clean cached repositories, fetch and check out the declared ref; + - if fetching/updating a cached repository fails, warn and reuse the cache; + - if the cached repository has local changes, warn and skip updates. + """ + + def __init__(self, cache_root: Path | str = DEFAULT_GIT_ASSET_CACHE_ROOT) -> None: + self.cache_root = Path(cache_root).expanduser() + self._sources_root = self.cache_root / "sources" + self._locks_root = self.cache_root / "locks" + + def resolve(self, repo_url: str, ref: str) -> GitAssetCheckout: + """Return a local checkout for `repo_url` at `ref`.""" + key = self._source_key(repo_url, ref) + checkout_path = self._sources_root / key / self._repo_slug(repo_url) + lock_path = self._locks_root / f"{key}.lock" + + self._sources_root.mkdir(parents=True, exist_ok=True) + self._locks_root.mkdir(parents=True, exist_ok=True) + + with FileLock(str(lock_path)): + if not checkout_path.exists(): + return self._clone_missing(repo_url, ref, checkout_path) + return self._refresh_cached(repo_url, ref, checkout_path) + + @staticmethod + def _source_key(repo_url: str, ref: str) -> str: + return sha256(f"{repo_url}\0{ref}".encode()).hexdigest()[:16] + + @staticmethod + def _repo_slug(repo_url: str) -> str: + parsed_path = urlparse(repo_url).path or repo_url + slug = Path(parsed_path.rstrip("/")).name + if slug.endswith(".git"): + slug = slug[:-4] + slug = "".join(char if char.isalnum() or char in {"-", "_", "."} else "-" for char in slug) + return slug or "checkout" + + def _clone_missing(self, repo_url: str, ref: str, checkout_path: Path) -> GitAssetCheckout: + temp_parent = checkout_path.parent + temp_parent.mkdir(parents=True, exist_ok=True) + temp_path = Path(tempfile.mkdtemp(prefix=f".{checkout_path.name}-", dir=temp_parent)) + try: + repo = Repo.clone_from(repo_url, temp_path) + self._checkout_ref(repo, ref) + temp_path.rename(checkout_path) + return GitAssetCheckout(path=checkout_path, repo_url=repo_url, ref=ref, updated=True) + except Exception as exc: + shutil.rmtree(temp_path, ignore_errors=True) + raise GitAssetCacheError( + f"Failed to fetch Git asset source {repo_url!r} at ref {ref!r}: {exc}" + ) from exc + + def _refresh_cached(self, repo_url: str, ref: str, checkout_path: Path) -> GitAssetCheckout: + try: + repo = Repo(checkout_path) + except (InvalidGitRepositoryError, NoSuchPathError) as exc: + raise GitAssetCacheError( + f"Cached asset path {checkout_path} is not a valid Git repository" + ) from exc + + if self._is_dirty(repo): + warnings.warn( + f"Git asset cache {checkout_path} has local changes; skipping upstream update.", + GitAssetCacheWarning, + stacklevel=2, + ) + return GitAssetCheckout( + path=checkout_path, + repo_url=repo_url, + ref=ref, + skipped_dirty_update=True, + ) + + try: + repo.remotes.origin.fetch(tags=True) + before = repo.head.commit.hexsha if repo.head.is_valid() else None + self._checkout_ref(repo, ref) + after = repo.head.commit.hexsha if repo.head.is_valid() else None + return GitAssetCheckout( + path=checkout_path, repo_url=repo_url, ref=ref, updated=before != after + ) + except Exception as exc: + warnings.warn( + f"Could not update Git asset cache {checkout_path}; using cached checkout: {exc}", + GitAssetCacheWarning, + stacklevel=2, + ) + return GitAssetCheckout( + path=checkout_path, + repo_url=repo_url, + ref=ref, + used_cached_fallback=True, + ) + + @staticmethod + def _is_dirty(repo: Repo) -> bool: + return repo.is_dirty(untracked_files=True) + + @staticmethod + def _checkout_ref(repo: Repo, ref: str) -> None: + """Check out a branch, tag, or commit and update clean worktrees safely.""" + remote_ref = f"origin/{ref}" + remote_refs = {str(r) for r in repo.remotes.origin.refs} + + if remote_ref in remote_refs: + repo.git.checkout("-B", ref, remote_ref) + repo.git.reset("--hard", remote_ref) + return + + try: + repo.git.checkout(ref) + except GitCommandError as exc: + raise GitAssetCacheError(f"Could not check out ref {ref!r}") from exc diff --git a/dimos/robot/assets/manager.py b/dimos/robot/assets/manager.py new file mode 100644 index 0000000000..3195a91639 --- /dev/null +++ b/dimos/robot/assets/manager.py @@ -0,0 +1,294 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Robot model asset declarations and lazy path adapters.""" + +from __future__ import annotations + +from collections.abc import Mapping +from dataclasses import dataclass, field +from enum import Enum +from pathlib import Path + +from dimos.robot.assets.git_cache import GitAssetCache + + +class RobotAssetError(RuntimeError): + """Raised when a robot asset declaration cannot satisfy a request.""" + + +class ArtifactRole(str, Enum): + """Common robot asset artifact roles. + + Strings are canonical internally; this enum is a convenience for common roles. + """ + + URDF = "urdf" + MJCF = "mjcf" + SRDF = "srdf" + MESH_DIR = "mesh_dir" + + +@dataclass(frozen=True) +class RobotAssetDeclaration: + """Typed declaration for one robot model's assets.""" + + model: str + repo_url: str + ref: str + artifacts: Mapping[str, str] + package_roots: Mapping[str, str] = field(default_factory=dict) + xacro_args: Mapping[str, str] = field(default_factory=dict) + source_name: str | None = None + license: str | None = None + + +class RobotAssetManager: + """Resolve robot model artifacts and package roots from declarations.""" + + def __init__( + self, + declarations: Mapping[str, RobotAssetDeclaration] | None = None, + git_cache: GitAssetCache | None = None, + ) -> None: + self._declarations = dict(declarations or {}) + self._git_cache = git_cache or GitAssetCache() + + def get_declaration(self, model: str) -> RobotAssetDeclaration: + try: + return self._declarations[model] + except KeyError as exc: + available = ", ".join(sorted(self._declarations)) or "none" + raise RobotAssetError( + f"Unknown robot asset model {model!r}. Available models: {available}." + ) from exc + + def resolve_artifact(self, model: str, role: str | ArtifactRole) -> Path: + declaration = self.get_declaration(model) + role_key = _role_key(role) + try: + relative_path = declaration.artifacts[role_key] + except KeyError as exc: + available = ", ".join(sorted(declaration.artifacts)) or "none" + raise RobotAssetError( + f"Robot asset model {model!r} does not declare artifact role {role_key!r}. " + f"Available roles: {available}." + ) from exc + + path = self._checkout(declaration) / relative_path + if not path.exists(): + raise RobotAssetError( + f"Declared artifact {role_key!r} for robot asset model {model!r} does not exist: {path}" + ) + return path + + def resolve_package_root(self, model: str, package_name: str) -> Path: + declaration = self.get_declaration(model) + try: + relative_path = declaration.package_roots[package_name] + except KeyError as exc: + available = ", ".join(sorted(declaration.package_roots)) or "none" + raise RobotAssetError( + f"Robot asset model {model!r} does not declare ROS package root " + f"{package_name!r}. Available package roots: {available}." + ) from exc + + path = self._checkout(declaration) / relative_path + if not path.exists(): + raise RobotAssetError( + f"Declared package root {package_name!r} for robot asset model {model!r} " + f"does not exist: {path}" + ) + return path + + def package_roots(self, model: str) -> dict[str, Path]: + declaration = self.get_declaration(model) + return { + package_name: RobotAssetPackagePath(model, package_name, manager=self) + for package_name in declaration.package_roots + } + + def xacro_args(self, model: str) -> dict[str, str]: + return dict(self.get_declaration(model).xacro_args) + + def _checkout(self, declaration: RobotAssetDeclaration) -> Path: + return self._git_cache.resolve(declaration.repo_url, declaration.ref).path + + +class RobotAssetPath(type(Path())): # type: ignore[misc] + """Lazy Path-like adapter for a declared robot model artifact.""" + + def __new__( + cls, + model: str, + role: str | ArtifactRole, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> RobotAssetPath: + instance: RobotAssetPath = super().__new__(cls, ".") + object.__setattr__(instance, "_robot_asset_model", model) + object.__setattr__(instance, "_robot_asset_role", _role_key(role)) + object.__setattr__( + instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) + ) + object.__setattr__( + instance, "_robot_asset_manager", manager or default_robot_asset_manager() + ) + object.__setattr__(instance, "_robot_asset_resolved_cache", None) + return instance + + def __init__( + self, + model: str, + role: str | ArtifactRole, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> None: + del model, role, relative_parts, manager + + def _resolve(self) -> Path: + cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") + if cache is None: + manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") + model = object.__getattribute__(self, "_robot_asset_model") + role = object.__getattribute__(self, "_robot_asset_role") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + cache = manager.resolve_artifact(model, role).joinpath(*relative_parts) + object.__setattr__(self, "_robot_asset_resolved_cache", cache) + return cache + + def __getattribute__(self, name: str) -> object: + try: + object.__getattribute__(self, "_robot_asset_model") + except AttributeError: + return object.__getattribute__(self, name) + + if name.startswith("_robot_asset_") or name in {"_resolve"}: + return object.__getattribute__(self, name) + + return getattr(object.__getattribute__(self, "_resolve")(), name) + + def __str__(self) -> str: + return str(self._resolve()) + + def __fspath__(self) -> str: + return str(self._resolve()) + + def __truediv__(self, other: object) -> RobotAssetPath: + model = object.__getattribute__(self, "_robot_asset_model") + role = object.__getattribute__(self, "_robot_asset_role") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + manager = object.__getattribute__(self, "_robot_asset_manager") + return RobotAssetPath(model, role, *relative_parts, other, manager=manager) + + +class RobotAssetPackagePath(type(Path())): # type: ignore[misc] + """Lazy Path-like adapter for a declared ROS package root.""" + + def __new__( + cls, + model: str, + package_name: str, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> RobotAssetPackagePath: + instance: RobotAssetPackagePath = super().__new__(cls, ".") + object.__setattr__(instance, "_robot_asset_model", model) + object.__setattr__(instance, "_robot_asset_package_name", package_name) + object.__setattr__( + instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) + ) + object.__setattr__( + instance, "_robot_asset_manager", manager or default_robot_asset_manager() + ) + object.__setattr__(instance, "_robot_asset_resolved_cache", None) + return instance + + def __init__( + self, + model: str, + package_name: str, + *relative_parts: object, + manager: RobotAssetManager | None = None, + ) -> None: + del model, package_name, relative_parts, manager + + def _resolve(self) -> Path: + cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") + if cache is None: + manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") + model = object.__getattribute__(self, "_robot_asset_model") + package_name = object.__getattribute__(self, "_robot_asset_package_name") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + cache = manager.resolve_package_root(model, package_name).joinpath(*relative_parts) + object.__setattr__(self, "_robot_asset_resolved_cache", cache) + return cache + + def __getattribute__(self, name: str) -> object: + try: + object.__getattribute__(self, "_robot_asset_model") + except AttributeError: + return object.__getattribute__(self, name) + + if name.startswith("_robot_asset_") or name in {"_resolve"}: + return object.__getattribute__(self, name) + + return getattr(object.__getattribute__(self, "_resolve")(), name) + + def __str__(self) -> str: + return str(self._resolve()) + + def __fspath__(self) -> str: + return str(self._resolve()) + + def __truediv__(self, other: object) -> RobotAssetPackagePath: + model = object.__getattribute__(self, "_robot_asset_model") + package_name = object.__getattribute__(self, "_robot_asset_package_name") + relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") + manager = object.__getattribute__(self, "_robot_asset_manager") + return RobotAssetPackagePath(model, package_name, *relative_parts, other, manager=manager) + + +_DEFAULT_MANAGER: RobotAssetManager | None = None + + +def default_robot_asset_manager() -> RobotAssetManager: + global _DEFAULT_MANAGER + if _DEFAULT_MANAGER is None: + from dimos.robot.assets.declarations import ROBOT_ASSETS + + _DEFAULT_MANAGER = RobotAssetManager(ROBOT_ASSETS) + return _DEFAULT_MANAGER + + +def set_default_robot_asset_manager(manager: RobotAssetManager | None) -> None: + """Override the process-default robot asset manager. + + Passing ``None`` clears the override and restores lazy construction from the + DimOS declarations module on the next default lookup. + """ + global _DEFAULT_MANAGER + _DEFAULT_MANAGER = manager + + +def robot_asset_package_paths(model: str) -> dict[str, Path]: + return default_robot_asset_manager().package_roots(model) + + +def robot_asset_xacro_args(model: str) -> dict[str, str]: + return default_robot_asset_manager().xacro_args(model) + + +def _role_key(role: str | ArtifactRole) -> str: + return role.value if isinstance(role, ArtifactRole) else str(role) diff --git a/dimos/robot/assets/processing.py b/dimos/robot/assets/processing.py new file mode 100644 index 0000000000..f96cc8c17f --- /dev/null +++ b/dimos/robot/assets/processing.py @@ -0,0 +1,152 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Universal robot asset rendering helpers.""" + +from __future__ import annotations + +from collections.abc import Mapping +import hashlib +import os +from pathlib import Path +import re +from typing import Literal + +from dimos.robot.assets.git_cache import DEFAULT_ROBOT_ASSET_CACHE_ROOT +from dimos.utils.logging_config import setup_logger + +logger = setup_logger() + +PackageUriMode = Literal["preserve", "absolute"] + +DERIVED_ASSET_CACHE_ROOT = DEFAULT_ROBOT_ASSET_CACHE_ROOT / "derived" +_RENDERED_URDF_CACHE_ROOT = DERIVED_ASSET_CACHE_ROOT / "rendered_urdfs" + + +def render_urdf( + urdf_path: Path | str | os.PathLike[str], + package_paths: Mapping[str, Path | str | os.PathLike[str]] | None = None, + xacro_args: Mapping[str, str] | None = None, + *, + package_uri_mode: PackageUriMode = "preserve", +) -> Path: + """Render a URDF or Xacro artifact into a cached plain URDF file. + + This is the universal robot-description stage: it expands Xacro with the + supplied ROS package roots and can optionally rewrite ``package://`` URIs to + absolute filesystem paths. Consumer-specific cleanup, such as Drake-only tag + stripping, belongs in the consumer adapter. + """ + if package_uri_mode not in ("preserve", "absolute"): + raise ValueError(f"Unsupported package URI mode: {package_uri_mode!r}") + + source_path = Path(os.fspath(urdf_path)) + resolved_package_paths = normalize_package_paths(package_paths or {}) + resolved_xacro_args = dict(xacro_args or {}) + + cache_key = _generate_render_key( + source_path, + resolved_package_paths, + resolved_xacro_args, + package_uri_mode, + ) + cache_path = _RENDERED_URDF_CACHE_ROOT / cache_key / source_path.stem + cache_path.mkdir(parents=True, exist_ok=True) + rendered_urdf = cache_path / f"{source_path.stem}.urdf" + + if rendered_urdf.exists(): + logger.debug(f"Using cached rendered URDF: {rendered_urdf}") + return rendered_urdf + + if source_path.suffix in (".xacro", ".urdf.xacro"): + urdf_content = _process_xacro(source_path, resolved_package_paths, resolved_xacro_args) + else: + urdf_content = source_path.read_text() + + if package_uri_mode == "absolute": + urdf_content = resolve_package_uris(urdf_content, resolved_package_paths) + + rendered_urdf.write_text(urdf_content) + logger.info(f"Rendered URDF cached at: {rendered_urdf}") + return rendered_urdf + + +def resolve_package_uris( + urdf_content: str, + package_paths: Mapping[str, Path | str | os.PathLike[str]], +) -> str: + """Rewrite ``package://`` URIs in URDF XML to absolute filesystem paths.""" + resolved_package_paths = normalize_package_paths(package_paths) + pattern = r"""package://([^/]+)/(.+?)(["'<>\s])""" + + def replace_uri(match: re.Match[str]) -> str: + pkg_name = match.group(1) + rel_path = match.group(2) + suffix = match.group(3) + + if pkg_name in resolved_package_paths: + full_path = resolved_package_paths[pkg_name] / rel_path + if full_path.exists(): + return f"{full_path}{suffix}" + logger.warning(f"File not found: {full_path}") + + return match.group(0) + + return re.sub(pattern, replace_uri, urdf_content) + + +def normalize_package_paths( + package_paths: Mapping[str, Path | str | os.PathLike[str]], +) -> dict[str, Path]: + return { + package_name: Path(os.fspath(package_path)).resolve() + for package_name, package_path in package_paths.items() + } + + +def _generate_render_key( + urdf_path: Path, + package_paths: Mapping[str, Path], + xacro_args: Mapping[str, str], + package_uri_mode: PackageUriMode, +) -> str: + processing_version = "urdf-render-v1" + mtime = urdf_path.stat().st_mtime if urdf_path.exists() else 0 + key_data = repr( + ( + processing_version, + str(urdf_path), + mtime, + sorted((name, str(path)) for name, path in package_paths.items()), + sorted(xacro_args.items()), + package_uri_mode, + ) + ) + return hashlib.sha256(key_data.encode()).hexdigest()[:16] + + +def _process_xacro( + xacro_path: Path, + package_paths: dict[str, Path], + xacro_args: dict[str, str], +) -> str: + try: + from dimos.utils.ament_prefix import process_xacro + except ImportError: + raise ImportError( + "xacro is required for processing .xacro files. " + "Install the manipulation extra: pip install dimos[manipulation]" + ) + + return process_xacro(xacro_path, package_paths, xacro_args) diff --git a/dimos/robot/assets/test_processing.py b/dimos/robot/assets/test_processing.py new file mode 100644 index 0000000000..cd600727ae --- /dev/null +++ b/dimos/robot/assets/test_processing.py @@ -0,0 +1,62 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + +from dimos.robot.assets import processing + + +def test_render_urdf_preserves_package_uris_by_default( + tmp_path: Path, + monkeypatch, +) -> None: + monkeypatch.setattr(processing, "_RENDERED_URDF_CACHE_ROOT", tmp_path / "rendered") + urdf = tmp_path / "robot.urdf" + urdf.write_text( + "" + "" + "" + ) + + rendered = processing.render_urdf(urdf, {"pkg": tmp_path / "pkg"}) + + assert rendered.is_relative_to(tmp_path / "rendered") + assert "package://pkg/meshes/link.stl" in rendered.read_text() + + +def test_render_urdf_can_rewrite_package_uris_to_absolute_paths( + tmp_path: Path, + monkeypatch, +) -> None: + monkeypatch.setattr(processing, "_RENDERED_URDF_CACHE_ROOT", tmp_path / "rendered") + package_root = tmp_path / "pkg" + mesh = package_root / "meshes" / "link.stl" + mesh.parent.mkdir(parents=True) + mesh.write_text("solid link\nendsolid link\n") + urdf = tmp_path / "robot.urdf" + urdf.write_text( + "" + "" + "" + ) + + rendered = processing.render_urdf( + urdf, + {"pkg": package_root}, + package_uri_mode="absolute", + ) + + rendered_text = rendered.read_text() + assert "package://" not in rendered_text + assert str(mesh) in rendered_text diff --git a/dimos/robot/catalog/a750.py b/dimos/robot/catalog/a750.py index 2f19ff7146..c513bb0a72 100644 --- a/dimos/robot/catalog/a750.py +++ b/dimos/robot/catalog/a750.py @@ -19,7 +19,7 @@ import math from typing import Any -from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath diff --git a/dimos/robot/catalog/piper.py b/dimos/robot/catalog/piper.py index af6e351b49..4424381ede 100644 --- a/dimos/robot/catalog/piper.py +++ b/dimos/robot/catalog/piper.py @@ -18,7 +18,7 @@ from typing import Any -from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath diff --git a/dimos/robot/catalog/ufactory.py b/dimos/robot/catalog/ufactory.py index c43a949f8b..e6fa2de772 100644 --- a/dimos/robot/catalog/ufactory.py +++ b/dimos/robot/catalog/ufactory.py @@ -18,7 +18,11 @@ from typing import Any -from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets import ( + RobotAssetPath, + robot_asset_package_paths, + robot_asset_xacro_args, +) from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath @@ -67,9 +71,7 @@ def xarm7( **overrides: Any, ) -> RobotConfig: """Create an xArm7 robot configuration.""" - xacro_args: dict[str, str] = { - "dof": "7", - "limited": "true", + xacro_args = robot_asset_xacro_args("xarm7") | { "attach_xyz": f"{x_offset} {y_offset} {z_offset}", "attach_rpy": f"0 {pitch} 0", } @@ -119,9 +121,7 @@ def xarm6( **overrides: Any, ) -> RobotConfig: """Create an xArm6 robot configuration.""" - xacro_args: dict[str, str] = { - "dof": "6", - "limited": "true", + xacro_args = robot_asset_xacro_args("xarm6") | { "attach_xyz": f"{x_offset} {y_offset} {z_offset}", "attach_rpy": f"0 {pitch} 0", } diff --git a/dimos/robot/robot_asset_declarations.py b/dimos/robot/robot_asset_declarations.py index cdbf8059ae..4e15bf236c 100644 --- a/dimos/robot/robot_asset_declarations.py +++ b/dimos/robot/robot_asset_declarations.py @@ -12,68 +12,18 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Robot asset declarations resolved by :mod:`dimos.robot.asset_manager`.""" +"""Compatibility exports for :mod:`dimos.robot.assets.declarations`.""" -from __future__ import annotations +from dimos.robot.assets.declarations import ( + A750_DESCRIPTION_REPO, + PIPER_DESCRIPTION_REPO, + ROBOT_ASSETS, + XARM_ROS2_REPO, +) -from dimos.robot.asset_manager import RobotAssetDeclaration - -XARM_ROS2_REPO = "https://github.com/xArm-Developer/xarm_ros2" -PIPER_DESCRIPTION_REPO = "https://github.com/agilexrobotics/agx_arm_urdf" -A750_DESCRIPTION_REPO = "https://github.com/adob/a750_description" - - -ROBOT_ASSETS: dict[str, RobotAssetDeclaration] = { - "xarm6": RobotAssetDeclaration( - model="xarm6", - repo_url=XARM_ROS2_REPO, - ref="humble", - artifacts={ - "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", - "mesh_dir": "xarm_description/meshes", - }, - package_roots={"xarm_description": "xarm_description"}, - xacro_args={"dof": "6", "limited": "true"}, - source_name="xarm_ros2", - ), - "xarm7": RobotAssetDeclaration( - model="xarm7", - repo_url=XARM_ROS2_REPO, - ref="humble", - artifacts={ - "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", - "mesh_dir": "xarm_description/meshes", - }, - package_roots={"xarm_description": "xarm_description"}, - xacro_args={"dof": "7", "limited": "true"}, - source_name="xarm_ros2", - ), - "piper": RobotAssetDeclaration( - model="piper", - repo_url=PIPER_DESCRIPTION_REPO, - ref="main", - artifacts={ - "urdf": "piper/urdf/piper_with_gripper_description.xacro", - "urdf_ik": "piper/urdf/piper_description.urdf", - "mesh_dir": "piper/meshes", - }, - # Upstream URDFs reference package://agx_arm_description/agx_arm_urdf/... - # and expect the checkout directory to be named agx_arm_urdf inside the - # package root. GitAssetCache preserves that checkout directory name. - package_roots={"agx_arm_description": ".."}, - source_name="agx_arm_urdf", - license="MIT", - ), - "a750": RobotAssetDeclaration( - model="a750", - repo_url=A750_DESCRIPTION_REPO, - ref="master", - artifacts={ - "urdf": "urdf/a750_rev1.urdf", - "mesh_dir": "meshes/a750_rev1", - }, - package_roots={"a750_description": "."}, - source_name="a750_description", - license="MIT", - ), -} +__all__ = [ + "A750_DESCRIPTION_REPO", + "PIPER_DESCRIPTION_REPO", + "ROBOT_ASSETS", + "XARM_ROS2_REPO", +] diff --git a/dimos/robot/test_asset_manager.py b/dimos/robot/test_asset_manager.py index d9268beaa9..3216fef9ed 100644 --- a/dimos/robot/test_asset_manager.py +++ b/dimos/robot/test_asset_manager.py @@ -18,15 +18,19 @@ import pytest -from dimos.robot.asset_manager import ( +from dimos.robot.assets import ( ArtifactRole, RobotAssetDeclaration, RobotAssetError, RobotAssetManager, RobotAssetPackagePath, RobotAssetPath, + robot_asset_package_paths, + robot_asset_xacro_args, + set_default_robot_asset_manager, ) -from dimos.utils.git_asset_cache import GitAssetCache +from dimos.robot.assets.declarations import ROBOT_ASSETS +from dimos.robot.assets.git_cache import GitAssetCache def _git(cwd: Path, *args: str) -> str: @@ -100,3 +104,32 @@ def test_lazy_asset_paths_resolve_only_on_path_operations(asset_manager: RobotAs assert artifact_path.exists() assert (package_path / "package.xml").exists() + + +def test_default_manager_can_be_injected(asset_manager: RobotAssetManager) -> None: + set_default_robot_asset_manager(asset_manager) + try: + assert robot_asset_package_paths("testbot")["testbot_description"].exists() + assert robot_asset_xacro_args("testbot") == {} + finally: + set_default_robot_asset_manager(None) + + +def test_robot_asset_declarations_are_static_and_consistent() -> None: + known_roles = {role.value for role in ArtifactRole} | {"urdf_ik"} + + for key, declaration in ROBOT_ASSETS.items(): + assert key == declaration.model + assert declaration.repo_url + assert declaration.ref + assert declaration.artifacts + + for role, relative_path in declaration.artifacts.items(): + assert role in known_roles + assert relative_path + assert not Path(relative_path).is_absolute() + + for package_name, relative_path in declaration.package_roots.items(): + assert package_name + assert relative_path + assert not Path(relative_path).is_absolute() diff --git a/dimos/utils/git_asset_cache.py b/dimos/utils/git_asset_cache.py index 0732c27333..1b9d79651a 100644 --- a/dimos/utils/git_asset_cache.py +++ b/dimos/utils/git_asset_cache.py @@ -12,160 +12,22 @@ # See the License for the specific language governing permissions and # limitations under the License. -"""Git-backed cache for robot description sources and other assets.""" - -from __future__ import annotations - -from dataclasses import dataclass -from hashlib import sha256 -from pathlib import Path -import shutil -import tempfile -from urllib.parse import urlparse -import warnings - -from filelock import FileLock -from git import GitCommandError, InvalidGitRepositoryError, NoSuchPathError, Repo - -DEFAULT_ROBOT_ASSET_CACHE_ROOT = Path.home() / ".cache" / "dimos" / "robot_assets" - - -class GitAssetCacheError(RuntimeError): - """Raised when an asset source cannot be resolved.""" - - -class GitAssetCacheWarning(RuntimeWarning): - """Warning emitted when a cached checkout is usable but not fresh.""" - - -@dataclass(frozen=True) -class GitAssetCheckout: - """Resolved local checkout information.""" - - path: Path - repo_url: str - ref: str - updated: bool = False - used_cached_fallback: bool = False - skipped_dirty_update: bool = False - - -class GitAssetCache: - """Resolve `(repo_url, ref)` pairs into fresh-when-safe cached checkouts. - - Policy: - - clone when the cache is missing; - - for clean cached repositories, fetch and check out the declared ref; - - if fetching/updating a cached repository fails, warn and reuse the cache; - - if the cached repository has local changes, warn and skip updates. - """ - - def __init__(self, cache_root: Path | str = DEFAULT_ROBOT_ASSET_CACHE_ROOT) -> None: - self.cache_root = Path(cache_root).expanduser() - self._sources_root = self.cache_root / "sources" - self._locks_root = self.cache_root / "locks" - - def resolve(self, repo_url: str, ref: str) -> GitAssetCheckout: - """Return a local checkout for `repo_url` at `ref`.""" - key = self._source_key(repo_url, ref) - checkout_path = self._sources_root / key / self._repo_slug(repo_url) - lock_path = self._locks_root / f"{key}.lock" - - self._sources_root.mkdir(parents=True, exist_ok=True) - self._locks_root.mkdir(parents=True, exist_ok=True) - - with FileLock(str(lock_path)): - if not checkout_path.exists(): - return self._clone_missing(repo_url, ref, checkout_path) - return self._refresh_cached(repo_url, ref, checkout_path) - - @staticmethod - def _source_key(repo_url: str, ref: str) -> str: - digest = sha256(f"{repo_url}\0{ref}".encode()).hexdigest()[:16] - return digest - - @staticmethod - def _repo_slug(repo_url: str) -> str: - parsed_path = urlparse(repo_url).path or repo_url - slug = Path(parsed_path.rstrip("/")).name - if slug.endswith(".git"): - slug = slug[:-4] - slug = "".join(char if char.isalnum() or char in {"-", "_", "."} else "-" for char in slug) - return slug or "checkout" - - def _clone_missing(self, repo_url: str, ref: str, checkout_path: Path) -> GitAssetCheckout: - temp_parent = checkout_path.parent - temp_parent.mkdir(parents=True, exist_ok=True) - temp_path = Path(tempfile.mkdtemp(prefix=f".{checkout_path.name}-", dir=temp_parent)) - try: - repo = Repo.clone_from(repo_url, temp_path) - self._checkout_ref(repo, ref) - temp_path.rename(checkout_path) - return GitAssetCheckout(path=checkout_path, repo_url=repo_url, ref=ref, updated=True) - except Exception as exc: - shutil.rmtree(temp_path, ignore_errors=True) - raise GitAssetCacheError( - f"Failed to fetch robot asset source {repo_url!r} at ref {ref!r}: {exc}" - ) from exc - - def _refresh_cached(self, repo_url: str, ref: str, checkout_path: Path) -> GitAssetCheckout: - try: - repo = Repo(checkout_path) - except (InvalidGitRepositoryError, NoSuchPathError) as exc: - raise GitAssetCacheError( - f"Cached asset path {checkout_path} is not a valid Git repository" - ) from exc - - if self._is_dirty(repo): - warnings.warn( - f"Robot asset cache {checkout_path} has local changes; skipping upstream update.", - GitAssetCacheWarning, - stacklevel=2, - ) - return GitAssetCheckout( - path=checkout_path, - repo_url=repo_url, - ref=ref, - skipped_dirty_update=True, - ) - - try: - repo.remotes.origin.fetch(tags=True) - before = repo.head.commit.hexsha if repo.head.is_valid() else None - self._checkout_ref(repo, ref) - after = repo.head.commit.hexsha if repo.head.is_valid() else None - return GitAssetCheckout( - path=checkout_path, repo_url=repo_url, ref=ref, updated=before != after - ) - except Exception as exc: - warnings.warn( - f"Could not update robot asset cache {checkout_path}; using cached checkout: {exc}", - GitAssetCacheWarning, - stacklevel=2, - ) - return GitAssetCheckout( - path=checkout_path, - repo_url=repo_url, - ref=ref, - used_cached_fallback=True, - ) - - @staticmethod - def _is_dirty(repo: Repo) -> bool: - return repo.is_dirty(untracked_files=True) - - @staticmethod - def _checkout_ref(repo: Repo, ref: str) -> None: - """Check out a branch, tag, or commit and update clean worktrees safely.""" - remote_ref = f"origin/{ref}" - remote_refs = {str(r) for r in repo.remotes.origin.refs} - - if remote_ref in remote_refs: - repo.git.checkout("-B", ref, remote_ref) - repo.git.reset("--hard", remote_ref) - return - - try: - repo.git.checkout(ref) - except GitCommandError as exc: - raise GitAssetCacheError(f"Could not check out ref {ref!r}") from exc +"""Compatibility exports for :mod:`dimos.robot.assets.git_cache`.""" + +from dimos.robot.assets.git_cache import ( + DEFAULT_GIT_ASSET_CACHE_ROOT, + DEFAULT_ROBOT_ASSET_CACHE_ROOT, + GitAssetCache, + GitAssetCacheError, + GitAssetCacheWarning, + GitAssetCheckout, +) + +__all__ = [ + "DEFAULT_GIT_ASSET_CACHE_ROOT", + "DEFAULT_ROBOT_ASSET_CACHE_ROOT", + "GitAssetCache", + "GitAssetCacheError", + "GitAssetCacheWarning", + "GitAssetCheckout", +] diff --git a/dimos/utils/test_git_asset_cache.py b/dimos/utils/test_git_asset_cache.py index 4905ed9c3c..b8a01e1c90 100644 --- a/dimos/utils/test_git_asset_cache.py +++ b/dimos/utils/test_git_asset_cache.py @@ -18,7 +18,7 @@ import pytest -from dimos.utils.git_asset_cache import GitAssetCache, GitAssetCacheWarning +from dimos.robot.assets.git_cache import GitAssetCache, GitAssetCacheWarning def _git(cwd: Path, *args: str) -> str: From 58bbd0d3d8c0e453d50c3c6026e7d4dcf012f1cc Mon Sep 17 00:00:00 2001 From: cc Date: Tue, 16 Jun 2026 09:44:15 -0700 Subject: [PATCH 05/15] fix: unblock robot asset CI --- .../planning/utils/test_mesh_utils.py | 2 +- .../manipulation/test_manipulation_module.py | 2 +- dimos/robot/assets/__init__.py | 65 ------------------- dimos/robot/assets/test_processing.py | 2 +- dimos/robot/catalog/a750.py | 2 +- dimos/robot/catalog/piper.py | 2 +- dimos/robot/catalog/ufactory.py | 2 +- dimos/robot/test_all_blueprints.py | 8 +++ dimos/robot/test_asset_manager.py | 6 +- 9 files changed, 17 insertions(+), 74 deletions(-) delete mode 100644 dimos/robot/assets/__init__.py diff --git a/dimos/manipulation/planning/utils/test_mesh_utils.py b/dimos/manipulation/planning/utils/test_mesh_utils.py index 285b7a3115..d45cf2975e 100644 --- a/dimos/manipulation/planning/utils/test_mesh_utils.py +++ b/dimos/manipulation/planning/utils/test_mesh_utils.py @@ -15,7 +15,7 @@ from pathlib import Path from dimos.manipulation.planning.utils import mesh_utils -from dimos.robot.assets import processing +import dimos.robot.assets.processing as processing def test_prepare_urdf_for_drake_uses_rendered_urdf_and_keeps_drake_cleanup( diff --git a/dimos/manipulation/test_manipulation_module.py b/dimos/manipulation/test_manipulation_module.py index d2a886d63e..5d262037c1 100644 --- a/dimos/manipulation/test_manipulation_module.py +++ b/dimos/manipulation/test_manipulation_module.py @@ -36,7 +36,7 @@ from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 from dimos.msgs.sensor_msgs.JointState import JointState -from dimos.robot.assets import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths pytestmark = pytest.mark.self_hosted diff --git a/dimos/robot/assets/__init__.py b/dimos/robot/assets/__init__.py deleted file mode 100644 index e546727a31..0000000000 --- a/dimos/robot/assets/__init__.py +++ /dev/null @@ -1,65 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Git-backed robot asset resolution and universal asset processing.""" - -from dimos.robot.assets.git_cache import ( - DEFAULT_GIT_ASSET_CACHE_ROOT, - DEFAULT_ROBOT_ASSET_CACHE_ROOT, - GitAssetCache, - GitAssetCacheError, - GitAssetCacheWarning, - GitAssetCheckout, -) -from dimos.robot.assets.manager import ( - ArtifactRole, - RobotAssetDeclaration, - RobotAssetError, - RobotAssetManager, - RobotAssetPackagePath, - RobotAssetPath, - default_robot_asset_manager, - robot_asset_package_paths, - robot_asset_xacro_args, - set_default_robot_asset_manager, -) -from dimos.robot.assets.processing import ( - DERIVED_ASSET_CACHE_ROOT, - PackageUriMode, - render_urdf, - resolve_package_uris, -) - -__all__ = [ - "DEFAULT_GIT_ASSET_CACHE_ROOT", - "DEFAULT_ROBOT_ASSET_CACHE_ROOT", - "DERIVED_ASSET_CACHE_ROOT", - "ArtifactRole", - "GitAssetCache", - "GitAssetCacheError", - "GitAssetCacheWarning", - "GitAssetCheckout", - "PackageUriMode", - "RobotAssetDeclaration", - "RobotAssetError", - "RobotAssetManager", - "RobotAssetPackagePath", - "RobotAssetPath", - "default_robot_asset_manager", - "render_urdf", - "resolve_package_uris", - "robot_asset_package_paths", - "robot_asset_xacro_args", - "set_default_robot_asset_manager", -] diff --git a/dimos/robot/assets/test_processing.py b/dimos/robot/assets/test_processing.py index cd600727ae..798dcd556e 100644 --- a/dimos/robot/assets/test_processing.py +++ b/dimos/robot/assets/test_processing.py @@ -14,7 +14,7 @@ from pathlib import Path -from dimos.robot.assets import processing +import dimos.robot.assets.processing as processing def test_render_urdf_preserves_package_uris_by_default( diff --git a/dimos/robot/catalog/a750.py b/dimos/robot/catalog/a750.py index c513bb0a72..36ff72f6d4 100644 --- a/dimos/robot/catalog/a750.py +++ b/dimos/robot/catalog/a750.py @@ -19,7 +19,7 @@ import math from typing import Any -from dimos.robot.assets import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath diff --git a/dimos/robot/catalog/piper.py b/dimos/robot/catalog/piper.py index 4424381ede..17fd283a76 100644 --- a/dimos/robot/catalog/piper.py +++ b/dimos/robot/catalog/piper.py @@ -18,7 +18,7 @@ from typing import Any -from dimos.robot.assets import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath diff --git a/dimos/robot/catalog/ufactory.py b/dimos/robot/catalog/ufactory.py index e6fa2de772..26787b9b95 100644 --- a/dimos/robot/catalog/ufactory.py +++ b/dimos/robot/catalog/ufactory.py @@ -18,7 +18,7 @@ from typing import Any -from dimos.robot.assets import ( +from dimos.robot.assets.manager import ( RobotAssetPath, robot_asset_package_paths, robot_asset_xacro_args, diff --git a/dimos/robot/test_all_blueprints.py b/dimos/robot/test_all_blueprints.py index cdf72e9b6b..66724b6a3d 100644 --- a/dimos/robot/test_all_blueprints.py +++ b/dimos/robot/test_all_blueprints.py @@ -40,6 +40,10 @@ "coordinator-mobile-manip-mock", "coordinator-mock", "coordinator-mock-twist-base", + "coordinator-openarm-bimanual", + "coordinator-openarm-left", + "coordinator-openarm-mock", + "coordinator-openarm-right", "coordinator-piper", "coordinator-servo-xarm6", "coordinator-teleop-dual", @@ -50,6 +54,10 @@ "coordinator-xarm6", "coordinator-xarm7", "dual-xarm6-planner", + "keyboard-teleop-openarm", + "keyboard-teleop-openarm-mock", + "openarm-mock-planner-coordinator", + "openarm-planner-coordinator", "teleop-hosted-go2", "teleop-hosted-xarm7", "teleop-quest-dual", diff --git a/dimos/robot/test_asset_manager.py b/dimos/robot/test_asset_manager.py index 3216fef9ed..a231ba750a 100644 --- a/dimos/robot/test_asset_manager.py +++ b/dimos/robot/test_asset_manager.py @@ -18,7 +18,9 @@ import pytest -from dimos.robot.assets import ( +from dimos.robot.assets.declarations import ROBOT_ASSETS +from dimos.robot.assets.git_cache import GitAssetCache +from dimos.robot.assets.manager import ( ArtifactRole, RobotAssetDeclaration, RobotAssetError, @@ -29,8 +31,6 @@ robot_asset_xacro_args, set_default_robot_asset_manager, ) -from dimos.robot.assets.declarations import ROBOT_ASSETS -from dimos.robot.assets.git_cache import GitAssetCache def _git(cwd: Path, *args: str) -> str: From 7b921348766f9f9f5b99755b055dbe226e4ba6a9 Mon Sep 17 00:00:00 2001 From: cc Date: Tue, 16 Jun 2026 11:31:16 -0700 Subject: [PATCH 06/15] chore: cleanup leftover modules --- dimos/robot/asset_manager.py | 41 -------- dimos/robot/asset_processing.py | 31 ------ dimos/robot/assets/README.md | 95 +++++++++++++++++++ .../test_manager.py} | 0 dimos/robot/robot_asset_declarations.py | 29 ------ dimos/utils/git_asset_cache.py | 33 ------- .../manipulation/adding_a_custom_arm.md | 6 +- docs/capabilities/manipulation/readme.md | 2 +- 8 files changed, 99 insertions(+), 138 deletions(-) delete mode 100644 dimos/robot/asset_manager.py delete mode 100644 dimos/robot/asset_processing.py create mode 100644 dimos/robot/assets/README.md rename dimos/robot/{test_asset_manager.py => assets/test_manager.py} (100%) delete mode 100644 dimos/robot/robot_asset_declarations.py delete mode 100644 dimos/utils/git_asset_cache.py diff --git a/dimos/robot/asset_manager.py b/dimos/robot/asset_manager.py deleted file mode 100644 index d3f70176ee..0000000000 --- a/dimos/robot/asset_manager.py +++ /dev/null @@ -1,41 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Compatibility exports for :mod:`dimos.robot.assets.manager`.""" - -from dimos.robot.assets.manager import ( - ArtifactRole, - RobotAssetDeclaration, - RobotAssetError, - RobotAssetManager, - RobotAssetPackagePath, - RobotAssetPath, - default_robot_asset_manager, - robot_asset_package_paths, - robot_asset_xacro_args, - set_default_robot_asset_manager, -) - -__all__ = [ - "ArtifactRole", - "RobotAssetDeclaration", - "RobotAssetError", - "RobotAssetManager", - "RobotAssetPackagePath", - "RobotAssetPath", - "default_robot_asset_manager", - "robot_asset_package_paths", - "robot_asset_xacro_args", - "set_default_robot_asset_manager", -] diff --git a/dimos/robot/asset_processing.py b/dimos/robot/asset_processing.py deleted file mode 100644 index 3f1bedfab2..0000000000 --- a/dimos/robot/asset_processing.py +++ /dev/null @@ -1,31 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Compatibility exports for :mod:`dimos.robot.assets.processing`.""" - -from dimos.robot.assets.processing import ( - DERIVED_ASSET_CACHE_ROOT, - PackageUriMode, - normalize_package_paths, - render_urdf, - resolve_package_uris, -) - -__all__ = [ - "DERIVED_ASSET_CACHE_ROOT", - "PackageUriMode", - "normalize_package_paths", - "render_urdf", - "resolve_package_uris", -] diff --git a/dimos/robot/assets/README.md b/dimos/robot/assets/README.md new file mode 100644 index 0000000000..1dfbb9e0a8 --- /dev/null +++ b/dimos/robot/assets/README.md @@ -0,0 +1,95 @@ +# Robot Assets + +`dimos.robot.assets` resolves robot description sources into local filesystem paths. +It is the home for Git-backed robot model assets, package-root resolution, and +generic URDF rendering helpers. + +This directory is intentionally self-contained so it can be extracted later. Do +not add compatibility wrappers outside this module for new code. Import directly +from the source modules, for example: + +```python +from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +``` + +There is no `__init__.py` on purpose: DimOS disallows package `__init__.py` files +except at the root package to avoid accidental import side effects. + +## Cache behavior + +Assets live under: + +```text +~/.cache/dimos/robot_assets/ +├── sources/ # Git checkouts by source identity +├── locks/ # per-source file locks +└── derived/ + ├── rendered_urdfs/ # generic rendered URDF cache + └── drake_urdfs/ # Drake-specific prepared URDF cache +``` + +`GitAssetCache` uses the “fresh-when-safe” policy: + +- clone when the source is missing; +- update clean cached repos before use; +- warn and keep cached content if update fails; +- warn and skip update for dirty cached repos, preserving local edits. + +## Declaring a robot asset + +Add declarations in `declarations.py`: + +```python +from dimos.robot.assets.manager import RobotAssetDeclaration + +ROBOT_ASSETS["myarm"] = RobotAssetDeclaration( + model="myarm", + repo_url="https://github.com/example/myarm_description", + ref="main", # branch, tag, or commit + artifacts={ + "urdf": "urdf/myarm.urdf.xacro", + "mesh_dir": "meshes", + }, + package_roots={"myarm_description": "."}, + xacro_args={"limited": "true"}, +) +``` + +Artifact role keys are strings. Common roles are `urdf`, `mjcf`, `srdf`, and +`mesh_dir`; extra flat roles such as `urdf_ik` are allowed when a catalog needs +an additional model variant. + +`package_roots` maps ROS package names to directories inside the checkout. These +roots are used for `package://...` URIs and Xacro `$(find package_name)`. + +## Using assets in catalogs + +Catalogs should stay lazy at import time: + +```python +from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths + +model_path = RobotAssetPath("myarm", "urdf") +package_paths = robot_asset_package_paths("myarm") +``` + +`RobotAssetPath` and `RobotAssetPackagePath` defer clone/update/path validation +until path operations such as `str(path)`, `path.resolve()`, or `path.exists()`. + +## Rendering URDFs + +Use `processing.py` for generic robot-description rendering: + +```python +from dimos.robot.assets.processing import render_urdf + +rendered_path = render_urdf( + model_path, + package_paths, + xacro_args={"limited": "true"}, + package_uri_mode="preserve", # or "absolute" +) +``` + +Keep consumer-specific processing outside this module. For example, Drake-specific +cleanup still belongs in `dimos/manipulation/planning/utils/mesh_utils.py`. diff --git a/dimos/robot/test_asset_manager.py b/dimos/robot/assets/test_manager.py similarity index 100% rename from dimos/robot/test_asset_manager.py rename to dimos/robot/assets/test_manager.py diff --git a/dimos/robot/robot_asset_declarations.py b/dimos/robot/robot_asset_declarations.py deleted file mode 100644 index 4e15bf236c..0000000000 --- a/dimos/robot/robot_asset_declarations.py +++ /dev/null @@ -1,29 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Compatibility exports for :mod:`dimos.robot.assets.declarations`.""" - -from dimos.robot.assets.declarations import ( - A750_DESCRIPTION_REPO, - PIPER_DESCRIPTION_REPO, - ROBOT_ASSETS, - XARM_ROS2_REPO, -) - -__all__ = [ - "A750_DESCRIPTION_REPO", - "PIPER_DESCRIPTION_REPO", - "ROBOT_ASSETS", - "XARM_ROS2_REPO", -] diff --git a/dimos/utils/git_asset_cache.py b/dimos/utils/git_asset_cache.py deleted file mode 100644 index 1b9d79651a..0000000000 --- a/dimos/utils/git_asset_cache.py +++ /dev/null @@ -1,33 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Compatibility exports for :mod:`dimos.robot.assets.git_cache`.""" - -from dimos.robot.assets.git_cache import ( - DEFAULT_GIT_ASSET_CACHE_ROOT, - DEFAULT_ROBOT_ASSET_CACHE_ROOT, - GitAssetCache, - GitAssetCacheError, - GitAssetCacheWarning, - GitAssetCheckout, -) - -__all__ = [ - "DEFAULT_GIT_ASSET_CACHE_ROOT", - "DEFAULT_ROBOT_ASSET_CACHE_ROOT", - "GitAssetCache", - "GitAssetCacheError", - "GitAssetCacheWarning", - "GitAssetCheckout", -] diff --git a/docs/capabilities/manipulation/adding_a_custom_arm.md b/docs/capabilities/manipulation/adding_a_custom_arm.md index 31feb7c250..a5fc9979aa 100644 --- a/docs/capabilities/manipulation/adding_a_custom_arm.md +++ b/docs/capabilities/manipulation/adding_a_custom_arm.md @@ -503,19 +503,19 @@ If you want motion planning (collision-free trajectories via Drake), you need a ### 4a. Add your URDF -Prefer an upstream Robot Description Source and add a typed declaration in `dimos/robot/robot_asset_declarations.py`. `RobotAssetPath` is a lazy `Path`-like adapter: importing the catalog does not clone or update the repo, but the first concrete path access resolves the source into `~/.cache/dimos/robot_assets`. +Prefer an upstream Robot Description Source and add a typed declaration in `dimos/robot/assets/declarations.py`. `RobotAssetPath` is a lazy `Path`-like adapter: importing the catalog does not clone or update the repo, but the first concrete path access resolves the source into `~/.cache/dimos/robot_assets`. Use `LfsPath` only when the asset is intentionally vendored, locally modified, or has no suitable upstream source. ```python skip -from dimos.robot.asset_manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths from dimos.manipulation.manipulation_module import manipulation_module from dimos.manipulation.planning.spec import RobotModelConfig from dimos.msgs.geometry_msgs.PoseStamped import PoseStamped from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 -# Add a RobotAssetDeclaration for "yourarm" in robot_asset_declarations.py. +# Add a RobotAssetDeclaration for "yourarm" in dimos/robot/assets/declarations.py. # Common artifact roles are "urdf", "mjcf", "srdf", and "mesh_dir". _YOURARM_URDF_PATH = RobotAssetPath("yourarm", "urdf") diff --git a/docs/capabilities/manipulation/readme.md b/docs/capabilities/manipulation/readme.md index 4c4f258ab5..58796e8107 100644 --- a/docs/capabilities/manipulation/readme.md +++ b/docs/capabilities/manipulation/readme.md @@ -112,7 +112,7 @@ visualization backend. ## Robot model assets -XArm6, XArm7, Piper, and A-750 runtime model paths are resolved through the Robot Asset Manager (`dimos.robot.asset_manager`). Catalogs use `RobotAssetPath` so imports stay lightweight: no network or Git work happens until a concrete path is accessed. +XArm6, XArm7, Piper, and A-750 runtime model paths are resolved through the Robot Asset Manager (`dimos.robot.assets`). Catalogs use `RobotAssetPath` so imports stay lightweight: no network or Git work happens until a concrete path is accessed. Declared upstream sources are cached under `~/.cache/dimos/robot_assets`. The cache is fresh-when-safe: a missing checkout is cloned, a clean checkout is updated, update failures warn and continue with the cached copy, and dirty local checkouts are preserved with a warning. From 3e7b370f358c8b9c289a7eeabf737d06f846eb25 Mon Sep 17 00:00:00 2001 From: cc Date: Tue, 16 Jun 2026 11:48:06 -0700 Subject: [PATCH 07/15] chore: cleanup annoying os.fspath --- dimos/robot/config.py | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/dimos/robot/config.py b/dimos/robot/config.py index ae64b31c0d..c61929e1bf 100644 --- a/dimos/robot/config.py +++ b/dimos/robot/config.py @@ -21,7 +21,6 @@ from __future__ import annotations -import os from pathlib import Path from typing import Any @@ -222,15 +221,12 @@ def to_robot_model_config(self) -> RobotModelConfig: return RobotModelConfig( name=self.name, - model_path=Path(os.fspath(self.model_path)), + model_path=self.model_path, base_pose=base_pose, joint_names=joint_names, end_effector_link=self.end_effector_link, base_link=base_link, - package_paths={ - package_name: Path(os.fspath(package_path)) - for package_name, package_path in self.package_paths.items() - }, + package_paths=dict(self.package_paths), xacro_args=self.xacro_args, collision_exclusion_pairs=exclusions, auto_convert_meshes=self.auto_convert_meshes, From cdceb6979dedd0986220f8afe7355058d76167bb Mon Sep 17 00:00:00 2001 From: cc <55869557+TomCC7@users.noreply.github.com> Date: Tue, 16 Jun 2026 13:18:37 -0700 Subject: [PATCH 08/15] Update dimos/robot/assets/processing.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --- dimos/robot/assets/processing.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dimos/robot/assets/processing.py b/dimos/robot/assets/processing.py index f96cc8c17f..cb0ff942fc 100644 --- a/dimos/robot/assets/processing.py +++ b/dimos/robot/assets/processing.py @@ -69,7 +69,7 @@ def render_urdf( logger.debug(f"Using cached rendered URDF: {rendered_urdf}") return rendered_urdf - if source_path.suffix in (".xacro", ".urdf.xacro"): + if source_path.suffix == ".xacro": urdf_content = _process_xacro(source_path, resolved_package_paths, resolved_xacro_args) else: urdf_content = source_path.read_text() From 4059edd76a1da9d648fdf23bb58433cde022d78b Mon Sep 17 00:00:00 2001 From: cc Date: Tue, 16 Jun 2026 13:13:45 -0700 Subject: [PATCH 09/15] spec: remove --- .../.openspec.yaml | 2 - .../design.md | 87 ------------ .../migrate-robot-arms-to-git-assets/docs.md | 31 ----- .../proposal.md | 39 ------ .../specs/robot-asset-resolution/spec.md | 93 ------------- .../migrate-robot-arms-to-git-assets/tasks.md | 36 ----- openspec/config.yaml | 45 ------ openspec/schemas/dimos-capability/schema.yaml | 128 ------------------ .../dimos-capability/templates/design.md | 35 ----- .../dimos-capability/templates/docs.md | 19 --- .../dimos-capability/templates/proposal.md | 32 ----- .../dimos-capability/templates/spec.md | 16 --- .../dimos-capability/templates/tasks.md | 15 -- 13 files changed, 578 deletions(-) delete mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml delete mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/design.md delete mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/docs.md delete mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/proposal.md delete mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md delete mode 100644 openspec/changes/migrate-robot-arms-to-git-assets/tasks.md delete mode 100644 openspec/config.yaml delete mode 100644 openspec/schemas/dimos-capability/schema.yaml delete mode 100644 openspec/schemas/dimos-capability/templates/design.md delete mode 100644 openspec/schemas/dimos-capability/templates/docs.md delete mode 100644 openspec/schemas/dimos-capability/templates/proposal.md delete mode 100644 openspec/schemas/dimos-capability/templates/spec.md delete mode 100644 openspec/schemas/dimos-capability/templates/tasks.md diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml b/openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml deleted file mode 100644 index 40f4fff67f..0000000000 --- a/openspec/changes/migrate-robot-arms-to-git-assets/.openspec.yaml +++ /dev/null @@ -1,2 +0,0 @@ -schema: dimos-capability -created: 2026-06-16 diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/design.md b/openspec/changes/migrate-robot-arms-to-git-assets/design.md deleted file mode 100644 index b6827ee5db..0000000000 --- a/openspec/changes/migrate-robot-arms-to-git-assets/design.md +++ /dev/null @@ -1,87 +0,0 @@ -## Context - -DimOS manipulation robot catalogs currently declare model assets through `LfsPath` values that unpack archives from `data/.lfs`. The affected surfaces include `dimos/robot/catalog/ufactory.py`, `dimos/robot/catalog/piper.py`, `dimos/robot/catalog/a750.py`, manipulator blueprints, teleop/control blueprint wiring, `RobotConfig`, model parsing, Drake preprocessing, and manipulation docs/tests. - -The design decisions captured during exploration are recorded in `docs/adr/0001-git-backed-robot-asset-manager.md` and the glossary in `CONTEXT.md`. The change should migrate xArm, Piper, and A750, while leaving OpenArm on its current path because its description has local DimOS modifications. - -## Goals / Non-Goals - -**Goals:** - -- Resolve xArm, Piper, and A750 model artifacts from upstream robot description sources instead of copied LFS bundles. -- Preserve existing catalog ergonomics: consumers should still receive `Path`-compatible model paths and `dict[str, Path]` package roots. -- Avoid network activity at import time by using a lazy Path-like adapter for catalog constants. -- Support branch, tag, and commit refs, with commit pinning available but not mandatory. -- Use a standard user cache such as `~/.cache/dimos/robot_assets`. -- Update clean cached checkouts when upstream changes are available; warn and use cache when update fails; warn and skip updates for dirty cached checkouts. -- Keep Xacro processing and Drake URDF preparation in the existing parser/preparation layers. - -**Non-Goals:** - -- Do not migrate Unitree robot descriptions in this change. -- Do not migrate OpenArm in this change. -- Do not add local override machinery in v1; local tests can pass explicit `Path` values where needed. -- Do not introduce a YAML/TOML manifest requirement; typed Python declarations are the initial declaration format. -- Do not change CLI, skill/MCP, stream, or hardware command behavior. - -## DimOS Architecture - -Add a small asset layer under the DimOS runtime codebase, with two responsibilities: - -1. A generic Git cache component that resolves `(repo_url, ref)` into a local checkout path. -2. A robot-facing manager that resolves `(robot_model_name, artifact_role)` and ROS package roots from typed Python robot asset declarations. - -The core concepts are: - -- `GitAssetCache`: wraps Git operations and file locking. It clones missing repositories, checks whether cached repositories are dirty, fetches/checks out refs for clean repositories, and emits warnings/fallback reasons for update failures or dirty cache skips. -- `RobotAssetManager`: owns typed robot asset declarations and resolves artifact roles such as `urdf`, `mjcf`, `srdf`, `mesh_dir`, and extra string roles such as `urdf_ik`. -- `RobotAssetPath`: a lazy Path-like adapter used by catalog modules. It should not touch the network or filesystem at import time; it resolves only when a path operation requires a concrete filesystem path. -- Robot asset declarations: typed Python objects keyed by robot model name, containing a robot description source URL/ref, flat `artifacts: dict[str, str]`, `package_roots: dict[str, str]`, optional model-level `xacro_args`, and lightweight provenance fields such as source name/license when known. - -No new DimOS `Spec` Protocol is required unless implementation discovers an RPC/module boundary. This is a local library/API layer used by catalogs and planning/control consumers, not a stream transport or module interface. Existing blueprint composition should continue to consume catalog constants. No skills/MCP exposure or CLI entry point is required for v1. - -Package root handling must preserve existing parser behavior: declared ROS package roots are passed as `dict[str, Path]` to consumers that resolve `package://...` URIs and `$(find package_name)` expressions. Xacro files remain ordinary artifact paths; `dimos/robot/model_parser.py` and Drake preparation utilities remain responsible for Xacro expansion and mesh/package handling. - -Dependencies: add `GitPython` for Git operations and `filelock` for cross-process cache locking. Keep direct shelling out to Git behind the library only if GitPython cannot cover a required operation. - -## Decisions - -- Use a Git-backed Robot Asset Manager instead of Git LFS bundles for xArm, Piper, and A750. This removes copied upstream robot description bundles from the common DimOS maintenance path. -- Use a thin DimOS wrapper over GitPython and filelock rather than `robot_descriptions.py`. `robot_descriptions.py` is useful as a curated registry, but this change needs arbitrary upstream robot description sources and DimOS-specific freshness policy. -- Use robot-model-first declarations. Catalogs should ask for assets by robot model name and artifact role; the manager deduplicates source checkouts internally by `(repo_url, ref)`. -- Use flat string artifact roles. Common role constants may exist for `urdf`, `mjcf`, `srdf`, and `mesh_dir`, but strings are canonical internally and extra roles remain possible. -- Allow branch, tag, and commit refs. Branch/tag defaults optimize ease of use and freshness; commit refs remain available for CI, releases, or fragile assets. -- Use “fresh-when-safe” cache behavior: clone missing cache, update clean cached repositories, warn/use cache on update failure, and warn/skip update for dirty cached repositories. -- Keep Xacro processing outside asset resolution. Asset resolution returns paths and package roots; existing model parsing and Drake preparation layers expand Xacro and normalize URDF/mesh details. - -## Safety / Simulation / Replay - -This change must not alter robot commands, control loops, skills, or stream contracts. Safety risk is indirect: an incorrect model asset can affect planning, FK/IK, visualization, or simulation. Implementation should verify resolved model paths for xArm, Piper, and A750 through existing parsing/planning entry points before using them in robot-facing blueprints. - -Simulation and replay behavior should match current behavior after cache population. First-run network failures should fail clearly when no cached checkout exists. If a cached checkout exists and an upstream update check fails, DimOS should warn and continue with the cached copy. - -Manual QA should cover at least one xArm blueprint path and one non-xArm migrated arm path where practical, using simulation/replay or parser-level checks before any real hardware run. - -## Risks / Trade-offs - -- Upstream repository layouts may differ from current LFS bundle layouts. Mitigation: declare artifact paths and package roots explicitly per robot model and test them. -- Branch/tag refs reduce strict reproducibility compared with commit-only pinning. Mitigation: allow commit refs and document when to pin. -- Freshness checks add network dependency to first resolution. Mitigation: continue with existing cache when update fails, and fail fast only when no cache exists. -- Dirty cache preservation may leave developers on locally modified assets. Mitigation: warn clearly and never overwrite local cache edits automatically. -- Lazy Path behavior can expose edge cases if consumers expect concrete `pathlib.Path` internals. Mitigation: keep the adapter small, test common path operations, and cast to concrete paths at integration boundaries if needed. - -## Migration / Rollout - -1. Add Git cache and robot asset manager code with tests. -2. Add typed asset declarations for xArm, Piper, and A750. -3. Migrate catalog constants and `RobotConfig.package_paths` declarations for xArm first, then Piper and A750. -4. Update tests and docs that mention `LfsPath` as the canonical manipulator asset pattern. -5. Keep existing LFS archives until migrated paths are verified and rollback is no longer needed. -6. If any catalog exports or blueprint registry behavior changes, run `pytest dimos/robot/test_all_blueprints_generation.py`; otherwise no generated blueprint registry update is expected. - -Rollback is to restore affected catalog constants to `LfsPath` and keep the LFS archives in place. - -## Open Questions - -- Exact upstream robot description source URLs and refs for Piper and A750 need confirmation during implementation. -- Whether GitPython should be a core dependency or an optional extra depends on whether migrated manipulator catalogs are imported in minimal DimOS installs. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/docs.md b/openspec/changes/migrate-robot-arms-to-git-assets/docs.md deleted file mode 100644 index 5ebf2d4d7c..0000000000 --- a/openspec/changes/migrate-robot-arms-to-git-assets/docs.md +++ /dev/null @@ -1,31 +0,0 @@ -## User-Facing Docs - -- Update `docs/capabilities/manipulation/adding_a_custom_arm.md` so new custom arm guidance uses Robot Asset Manager declarations instead of presenting `LfsPath` bundles as the canonical path. -- Update manipulation capability docs that list model paths for xArm, Piper, and A750 so they describe robot asset declarations, artifact roles, ROS package roots, and cache behavior. -- Add or update a user-facing section under `docs/usage/` or `docs/capabilities/manipulation/` explaining: - - the Robot Asset Manager purpose, - - supported artifact roles (`urdf`, `mjcf`, `srdf`, `mesh_dir`, plus extra string roles), - - cache location and fresh-when-safe behavior, - - when commit refs should be used, - - why OpenArm remains on its current path for now. - -## Contributor Docs - -- Update contributor/development docs if implementation adds a new dependency or test workflow for robot assets. -- Document how to add a new typed robot asset declaration and how to choose package roots and artifact roles. -- Document test expectations for cache policy, dirty caches, update failures, and parser/planning compatibility. - -## Coding-Agent Docs - -- Update `AGENTS.md` or `docs/coding-agents/` only if the canonical workflow for adding robot model assets changes enough that coding agents need explicit instructions. -- Suggested coding-agent note: prefer Robot Asset Manager declarations for upstream robot description sources; use `LfsPath` only for assets that are intentionally vendored, locally modified, or not yet migrated. - -## Doc Validation - -- Run `uv run doclinks` after editing docs, if available in the environment. -- Run `md-babel-py run ` only for docs containing executable Python snippets. -- Run targeted tests referenced by docs if examples are made executable. - -## No Docs Needed - -Documentation changes are needed because this change replaces the documented LFS-based manipulator onboarding pattern and introduces user-visible cache/update behavior. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/proposal.md b/openspec/changes/migrate-robot-arms-to-git-assets/proposal.md deleted file mode 100644 index a622559259..0000000000 --- a/openspec/changes/migrate-robot-arms-to-git-assets/proposal.md +++ /dev/null @@ -1,39 +0,0 @@ -## Why - -DimOS currently keeps several manipulation robot description bundles in Git LFS archives under `data/.lfs`. That makes routine development depend on copied binary/source asset bundles inside this repository, increases repository-specific asset maintenance, and makes it harder to track upstream robot description changes for supported arms. - -This change migrates the xArm, Piper, and A750 manipulation robot model assets to a Git-backed Robot Asset Manager. DimOS should resolve robot description sources from upstream repositories into a standard user cache while preserving the existing model-path and package-path behavior used by planning, control, parsing, simulation, and documentation. - -## What Changes - -- Add a Git-backed robot asset resolution capability for selected manipulation robot model files and package roots. -- Add typed Python robot asset declarations for xArm, Piper, and A750, with robot-model-first lookup and internal source checkout deduplication. -- Add a lazy Path-like adapter so existing `RobotConfig.model_path` and `package_paths` consumers can keep receiving filesystem paths without network access at import time. -- Migrate xArm, Piper, and A750 catalog model paths and package roots away from LFS-backed `LfsPath` declarations where suitable upstream robot description sources are available. -- Keep OpenArm on its current path for now because it has local DimOS modifications. -- Preserve Xacro, package URI, `$(find ...)`, MJCF/SRDF, and mesh directory compatibility through existing parser and Drake preparation layers. -- No **BREAKING** CLI, skill/MCP, or hardware-safety behavior changes are intended. - -## Affected DimOS Surfaces - -- Modules/streams: manipulation planning and control modules that consume `RobotConfig.model_path`, FK/IK model paths, package paths, Xacro inputs, MJCF files, SRDF files, and mesh directories. -- Blueprints/CLI: xArm, Piper, and A750 manipulator blueprints and teleop/control blueprint wiring that reference catalog constants; no CLI command behavior changes are intended. -- Skills/MCP: no direct skill or MCP tool behavior changes are intended. -- Hardware/simulation/replay: xArm, Piper, and A750 real/sim manipulation stacks may resolve model assets from the user cache instead of LFS bundles; OpenArm remains unchanged. -- Docs/generated registries: manipulation docs and tests that describe LFS-backed onboarding or hardcoded model bundle paths need updates; no generated blueprint registry changes are expected unless catalog exports change. - -## Capabilities - -### New Capabilities - -- `robot-asset-resolution`: Resolving robot model artifacts and ROS package roots from upstream robot description sources into local filesystem paths for DimOS consumers. - -### Modified Capabilities - -- None. - -## Impact - -Developers get easier robot asset maintenance and less need to copy upstream description bundles into DimOS. First use of a migrated arm may require network access to populate `~/.cache/dimos/robot_assets`; later uses can continue from the cached copy if upstream update checks fail. Clean cached repositories update when upstream changes are available, while dirty cached repositories are preserved with warnings. - -Compatibility risk is concentrated around upstream repository layout differences, package path resolution, Xacro expansion, and FK/IK model constants. The change needs unit tests for cache/update policy and asset path resolution, plus integration coverage for xArm, Piper, and A750 catalog paths through existing model parsing/planning entry points. Documentation should explain the new asset declarations, cache behavior, supported artifact roles, and the temporary split where OpenArm remains LFS-backed. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md b/openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md deleted file mode 100644 index 3fd153ecda..0000000000 --- a/openspec/changes/migrate-robot-arms-to-git-assets/specs/robot-asset-resolution/spec.md +++ /dev/null @@ -1,93 +0,0 @@ -## ADDED Requirements - -### Requirement: Resolve robot model artifacts by role - -DimOS SHALL resolve declared robot model artifacts for supported manipulation robot models by robot model name and artifact role. - -#### Scenario: Resolve a migrated arm URDF -- **GIVEN** a migrated robot model such as xArm, Piper, or A750 has a declared `urdf` artifact role -- **WHEN** a DimOS catalog or model consumer requests that artifact -- **THEN** DimOS returns a local filesystem path to the declared URDF or Xacro file -- **AND** existing model parsing consumers can use the returned path as a normal path-like model input. - -#### Scenario: Resolve additional flat artifact roles -- **GIVEN** a migrated robot model declares artifact roles such as `mjcf`, `srdf`, `mesh_dir`, or an extra string role such as `urdf_ik` -- **WHEN** a consumer requests one of those declared roles -- **THEN** DimOS returns the local filesystem path for that role -- **AND** DimOS reports an explicit error when the requested role is not declared for that robot model. - -### Requirement: Resolve ROS package roots for model consumers - -DimOS SHALL provide ROS package root mappings for migrated robot models when their model files require package-based resource resolution. - -#### Scenario: Resolve package URI resources -- **GIVEN** a migrated robot model declares a ROS package root for a package used by `package://...` URIs -- **WHEN** a parser, planner, or Drake preparation layer receives the model path and package roots -- **THEN** the consumer can resolve package-based mesh and resource references using the declared package root -- **AND** DimOS preserves compatibility with existing `dict[str, Path]` package root consumers. - -#### Scenario: Resolve Xacro find expressions -- **GIVEN** a migrated robot model declares a ROS package root for a package used by `$(find package_name)` in Xacro -- **WHEN** existing Xacro processing expands the model file -- **THEN** the package name resolves to the declared local package root -- **AND** Xacro processing remains the responsibility of the existing parser or Drake preparation layer. - -### Requirement: Populate and reuse a standard robot asset cache - -DimOS SHALL store fetched robot description sources in a standard user cache and reuse cached sources across runs. - -#### Scenario: Cache is missing -- **GIVEN** a requested migrated robot model has no cached source checkout -- **WHEN** DimOS resolves one of its artifacts -- **THEN** DimOS fetches the declared robot description source into the standard robot asset cache -- **AND** DimOS fails with a clear error if the source cannot be fetched and no cached checkout exists. - -#### Scenario: Cache is present and update succeeds -- **GIVEN** a requested migrated robot model has a clean cached source checkout -- **WHEN** DimOS resolves one of its artifacts and the upstream source has changed for the declared ref -- **THEN** DimOS updates the cached checkout before returning the artifact path -- **AND** the returned path points into the updated cached checkout. - -#### Scenario: Cache is present and update fails -- **GIVEN** a requested migrated robot model has a cached source checkout -- **WHEN** DimOS cannot check for or apply an upstream update -- **THEN** DimOS warns about the update failure -- **AND** DimOS continues using the cached checkout. - -#### Scenario: Cache has local changes -- **GIVEN** a requested migrated robot model has a cached source checkout with local changes -- **WHEN** DimOS resolves one of its artifacts -- **THEN** DimOS warns that the cache has local changes and skips upstream update -- **AND** DimOS returns paths from the dirty cached checkout without overwriting local edits. - -### Requirement: Preserve catalog path compatibility - -DimOS SHALL expose migrated robot assets through catalog declarations that remain compatible with existing path-based robot model consumers. - -#### Scenario: Catalog import remains lightweight -- **GIVEN** a module imports xArm, Piper, or A750 catalog constants -- **WHEN** the import completes -- **THEN** DimOS does not fetch robot description sources during import -- **AND** any network or cache resolution work is deferred until a concrete filesystem path is needed. - -#### Scenario: Existing RobotConfig consumers receive compatible paths -- **GIVEN** a migrated catalog creates a DimOS Robot Model Config with a model path and package roots -- **WHEN** an existing planner, parser, simulation, or visualization consumer reads that config -- **THEN** the model path behaves as a path-like filesystem value -- **AND** package roots remain compatible with existing `dict[str, Path]` expectations. - -### Requirement: Support flexible source refs - -DimOS SHALL support branch, tag, and commit refs in robot asset declarations for migrated robot models. - -#### Scenario: Declaration uses branch or tag ref -- **GIVEN** a migrated robot model declaration uses an upstream branch or tag ref -- **WHEN** DimOS resolves an artifact for that model -- **THEN** DimOS fetches and checks out the declared ref according to the cache freshness policy -- **AND** developers can use upstream-moving refs when ease of use and freshness are preferred. - -#### Scenario: Declaration uses commit ref -- **GIVEN** a migrated robot model declaration uses a commit ref -- **WHEN** DimOS resolves an artifact for that model -- **THEN** DimOS checks out that commit in the cache -- **AND** releases, CI, or fragile assets can use pinned refs when stronger reproducibility is required. diff --git a/openspec/changes/migrate-robot-arms-to-git-assets/tasks.md b/openspec/changes/migrate-robot-arms-to-git-assets/tasks.md deleted file mode 100644 index eb7b22c397..0000000000 --- a/openspec/changes/migrate-robot-arms-to-git-assets/tasks.md +++ /dev/null @@ -1,36 +0,0 @@ -## 1. Implementation - -- [x] 1.1 Add runtime dependencies for Git-backed asset resolution (`GitPython` and `filelock`) in project dependency configuration. -- [x] 1.2 Implement a generic Git asset cache with clone-on-miss, clean-cache update, update-failure fallback, dirty-cache skip, and per-source file locking. -- [x] 1.3 Add unit tests for Git asset cache behavior, including missing cache failure, successful clean update, update failure with cached fallback, and dirty cache preservation. -- [x] 1.4 Implement typed robot asset declarations with robot-model-first lookup, flat artifact role strings, ROS package roots, optional model-level Xacro args, and source checkout deduplication by `(repo_url, ref)`. -- [x] 1.5 Implement `RobotAssetManager` resolution for artifact paths and package roots, including explicit errors for unknown robot models or undeclared artifact roles. -- [x] 1.6 Implement `RobotAssetPath` as a lazy Path-like catalog adapter that avoids network/filesystem resolution at import time and resolves only when a concrete path is needed. -- [x] 1.7 Add unit tests for robot asset declaration lookup, package root resolution, lazy import behavior, and common path-like operations used by existing consumers. -- [x] 1.8 Identify and verify upstream robot description source URLs, refs, artifact paths, and ROS package roots for xArm, Piper, and A750. -- [x] 1.9 Add xArm, Piper, and A750 robot asset declarations for `urdf`, any required FK/IK URDF role such as `urdf_ik`, `mjcf` where applicable, `srdf` where applicable, and `mesh_dir` where applicable. -- [x] 1.10 Migrate xArm catalog constants and package roots from `LfsPath` to robot asset declarations and `RobotAssetPath`. -- [x] 1.11 Migrate Piper catalog constants and package roots from `LfsPath` to robot asset declarations and `RobotAssetPath`. -- [x] 1.12 Migrate A750 catalog constants and package roots from `LfsPath` to robot asset declarations and `RobotAssetPath`. -- [x] 1.13 Keep OpenArm catalog behavior unchanged and add comments/tests where useful to make the intentional non-migration clear. -- [x] 1.14 Update tests or fixtures that directly reference `get_data("xarm_description")`, Piper LFS paths, or A750 LFS paths so they validate the new asset resolution path instead. -- [x] 1.15 If catalog exports, blueprint names, or generated registry inputs change, regenerate and verify the blueprint registry with `pytest dimos/robot/test_all_blueprints_generation.py`. - -## 2. Documentation - -- [x] 2.1 Update `docs/capabilities/manipulation/adding_a_custom_arm.md` to describe Robot Asset Manager declarations as the preferred upstream robot description source workflow. -- [x] 2.2 Update xArm, Piper, and A750 manipulation docs to describe resolved artifact roles, ROS package roots, cache behavior, and any changed model path examples. -- [x] 2.3 Add or update a user-facing Robot Asset Manager documentation section covering purpose, cache location, fresh-when-safe behavior, supported artifact roles, and branch/tag/commit ref guidance. -- [x] 2.4 Update contributor docs if new dependency, cache testing, or robot asset declaration workflow details need contributor guidance. -- [x] 2.5 Update `AGENTS.md` or `docs/coding-agents/` if the implementation changes the recommended coding-agent workflow for adding robot model assets. - -## 3. Verification - -- [x] 3.1 Run `openspec validate migrate-robot-arms-to-git-assets`. -- [x] 3.2 Run focused unit tests for the Git asset cache and robot asset manager. -- [x] 3.3 Run focused catalog/model parsing tests for xArm, Piper, and A750 migrated paths. -- [x] 3.4 Run existing manipulation planning/control tests affected by catalog model path changes, including the current `dimos/manipulation/test_manipulation_module.py` target if still applicable. -- [x] 3.5 Run `uv run pytest dimos/robot/test_all_blueprints_generation.py` if blueprint registry inputs or generated registry output may have changed. -- [x] 3.6 Run `uv run doclinks` after documentation updates. -- [x] 3.7 Run `md-babel-py run ` for any changed documentation file that contains executable Python snippets. -- [x] 3.8 Manually QA at least one migrated xArm path and one non-xArm migrated path through parser/planning or simulation/replay before any real hardware use. diff --git a/openspec/config.yaml b/openspec/config.yaml deleted file mode 100644 index 62a72bba63..0000000000 --- a/openspec/config.yaml +++ /dev/null @@ -1,45 +0,0 @@ -schema: dimos-capability - -context: | - DimOS is a robotics operating system for generalist robots. Modules communicate - through typed streams (`In[T]`, `Out[T]`) over LCM, SHM, ROS, DDS, or other - transports. Blueprints compose modules into runnable robot stacks. Skills are - `@skill`-annotated RPC methods exposed to agents and MCP clients. - - Terminology boundary: - - "OpenSpec spec" means a behavior specification under `openspec/specs/`. - - "DimOS Spec" means a Python Protocol/RPC contract in `*_spec.py` files, - usually inheriting `dimos.spec.utils.Spec` and `typing.Protocol`. - Keep these separate. OpenSpec specs describe observable behavior; DimOS Specs - describe code-level module interfaces. - - OpenSpec specs should capture current behavior, user/developer-visible - outcomes, public CLI/API/tool surfaces, robot safety constraints, and testable - scenarios. Put implementation choices, class names, module wiring, generated - registry updates, and rollout details in `design.md` or `tasks.md`. - - Documentation lives in: - - `docs/usage/` for user-facing concepts and APIs. - - `docs/capabilities/` for capability and platform guides. - - `docs/development/` for contributor process. - - `docs/coding-agents/` and `AGENTS.md` for coding-agent guidance. - -rules: - proposal: - - "Identify affected DimOS surfaces: modules, streams, blueprints, CLI, skills/MCP, docs, hardware, simulation, replay, or generated registries." - - Use capability names that match behavior domains, not Python class names. - - Mark hardware safety or public API/CLI changes explicitly. - specs: - - Write behavior-first requirements; avoid implementation detail unless it is externally observable. - - Every requirement must include at least one `#### Scenario:` block with concrete observable outcomes. - - Use "OpenSpec capability spec" when prose might otherwise be confused with DimOS Python `Spec` Protocols. - design: - - Call out DimOS `Spec` Protocols, adapter Protocols, blueprint composition, stream names/types, and skill/MCP exposure when relevant. - - Mention generated files and required regeneration commands, especially `pytest dimos/robot/test_all_blueprints_generation.py` for blueprint registry changes. - - Include hardware/simulation/replay assumptions and safety constraints for robot-facing work. - docs: - - List user-facing docs, contributor docs, coding-agent docs, and AGENTS.md updates required by the change. - - Include documentation validation commands for changed docs, such as `doclinks` and `md-babel-py run ` where applicable. - tasks: - - Include verification tasks for OpenSpec validation, relevant pytest targets, type checks when needed, and manual QA through the user-facing surface. - - Add registry generation tasks when blueprint names, module classes, or generated registry inputs change. diff --git a/openspec/schemas/dimos-capability/schema.yaml b/openspec/schemas/dimos-capability/schema.yaml deleted file mode 100644 index fedb7964ee..0000000000 --- a/openspec/schemas/dimos-capability/schema.yaml +++ /dev/null @@ -1,128 +0,0 @@ -name: dimos-capability -version: 1 -description: DimOS capability workflow - proposal → specs/design/docs → tasks -artifacts: - - id: proposal - generates: proposal.md - description: DimOS change proposal covering intent, scope, capability impact, and affected robot/software surfaces - template: proposal.md - instruction: | - Create the proposal document that establishes WHY this change is needed and what DimOS behavior it affects. - - Sections: - - **Why**: 1-2 concise paragraphs on the problem or opportunity. Explain why the change matters now. - - **What Changes**: Bullet list of added, modified, or removed behavior. Mark public API/CLI or hardware-safety breaking changes with **BREAKING**. - - **Affected DimOS Surfaces**: Identify modules, streams, blueprints, CLI commands, skills/MCP tools, docs, hardware, simulation, replay, generated registries, or external protocols touched by the change. - - **Capabilities**: Identify which OpenSpec capability specs will be created or modified: - - **New Capabilities**: List behavior domains introduced by the change. Each becomes `specs//spec.md`. Use kebab-case names (for example, `agent-skills-mcp`, `blueprint-composition`, `manipulation-stack`). - - **Modified Capabilities**: List existing `openspec/specs//` entries whose requirements change. Only include spec-level behavior changes, not implementation-only refactors. - - **Impact**: Summarize user/developer impact, compatibility risks, dependency changes, documentation updates, and test/QA scope. - - Keep proposals concise. Do not include line-by-line implementation details; put architecture and rollout decisions in `design.md`. - requires: [] - - id: specs - generates: specs/**/*.md - description: Behavior-first OpenSpec capability delta specifications - template: spec.md - instruction: | - Create OpenSpec capability specs that define WHAT DimOS should do, not how it is implemented. - - Create one delta spec file per capability listed in proposal.md: - - New capabilities: use `specs//spec.md` with the exact kebab-case name from the proposal. - - Modified capabilities: use the existing folder from `openspec/specs//`. - - Use these delta sections as `##` headers: - - **ADDED Requirements**: New externally observable behavior. - - **MODIFIED Requirements**: Changed behavior. Include the full updated requirement block, not a partial patch. - - **REMOVED Requirements**: Deprecated behavior. Include **Reason** and **Migration**. - - **RENAMED Requirements**: Name-only changes. Use FROM:/TO: format. - - Requirement format: - - Use `### Requirement: `. - - Use SHALL/MUST for normative requirements. - - Include at least one `#### Scenario: ` per requirement. Scenario headings MUST use exactly four `#` characters. - - Prefer `- **GIVEN**`, `- **WHEN**`, `- **THEN**`, and `- **AND**` bullets. - - Cover happy path plus meaningful edge/error/safety cases. - - DimOS-specific guidance: - - Specify user/developer-visible behavior, robot outcomes, CLI behavior, skill/MCP tool behavior, stream contracts, safety constraints, and compatibility expectations. - - Avoid Python class names, private module internals, transport implementation choices, and generated-file details unless those details are observable API contracts. - - Use "OpenSpec capability spec" in prose when needed to avoid confusion with DimOS Python `Spec` Protocols. - - If the behavior only changes implementation and not observable requirements, do not create a spec delta. - requires: - - proposal - - id: design - generates: design.md - description: DimOS technical design and architecture decisions - template: design.md - instruction: | - Create the design document that explains HOW the change should be implemented in DimOS. - - Include design.md for cross-module changes, new robot/hardware integration, new public interfaces, new dependencies, safety-sensitive behavior, generated registry changes, or unclear architecture. - - Sections: - - **Context**: Current state, relevant modules/blueprints/docs, and constraints. - - **Goals / Non-Goals**: What the design achieves and explicitly excludes. - - **DimOS Architecture**: Modules, streams, transports, blueprints, RPC/module refs, DimOS `Spec` Protocols, adapter Protocols, skills/MCP exposure, CLI entry points, and generated registries involved. - - **Decisions**: Key choices with rationale and alternatives considered. - - **Safety / Simulation / Replay**: Hardware assumptions, sim/replay behavior, safety constraints, and manual QA surface. - - **Risks / Trade-offs**: Known risks and mitigations. - - **Migration / Rollout**: Compatibility, generated files, docs, and deployment steps. - - **Open Questions**: Outstanding decisions or unknowns. - - Reference proposal.md for intent and specs for behavior. Keep line-by-line work in tasks.md. - requires: - - proposal - - id: docs - generates: docs.md - description: Documentation impact plan for user, contributor, and coding-agent docs - template: docs.md - instruction: | - Create the documentation impact plan for the change. - - Sections: - - **User-Facing Docs**: Updates under `docs/usage/`, `docs/capabilities/`, `docs/platforms/`, or README files. - - **Contributor Docs**: Updates under `docs/development/`. - - **Coding-Agent Docs**: Updates under `docs/coding-agents/` or `AGENTS.md`. - - **Doc Validation**: Commands needed for changed docs, such as `doclinks`, `md-babel-py run `, and `bin/gen-diagrams`. - - **No Docs Needed**: If no docs are needed, explain why. - - Match `docs/development/writing_docs.md`: contributor-only docs belong in `docs/development`; user-facing behavior belongs in `docs/usage` or `docs/capabilities`. - requires: - - proposal - - id: tasks - generates: tasks.md - description: Implementation, validation, docs, and manual-QA checklist - template: tasks.md - instruction: | - Create the implementation checklist. The apply phase parses checkbox format, so every actionable task MUST use `- [ ]`. - - Guidelines: - - Group tasks under numbered `##` headings. - - Each task must be `- [ ] X.Y Task description`. - - Keep tasks small enough to complete in one focused session. - - Order tasks by dependency. - - Include docs and validation tasks from docs.md. - - Include generated registry tasks when blueprints or module registry inputs change. - - Include manual QA through the actual user surface: CLI, TUI, HTTP API, MCP tool, simulation/replay blueprint, hardware procedure, or library driver. - - Typical DimOS validation tasks: - - Run `openspec validate `. - - Run focused pytest targets for changed modules. - - Run `pytest dimos/robot/test_all_blueprints_generation.py` when blueprint registry output may change. - - Run docs validation commands for changed docs. - - Run lints/types when the touched area requires them. - - Reference specs for WHAT, design for HOW, and docs.md for documentation work. - requires: - - specs - - design - - docs -apply: - requires: - - tasks - tracks: tasks.md - instruction: | - Read proposal.md, specs, design.md, docs.md, and tasks.md before editing code. - Work through pending tasks, mark checkboxes complete as they finish, and keep artifacts current when implementation changes the plan. - Verify with OpenSpec validation, focused tests, docs checks, and manual QA through the relevant DimOS surface. diff --git a/openspec/schemas/dimos-capability/templates/design.md b/openspec/schemas/dimos-capability/templates/design.md deleted file mode 100644 index 25031ceb8b..0000000000 --- a/openspec/schemas/dimos-capability/templates/design.md +++ /dev/null @@ -1,35 +0,0 @@ -## Context - - - -## Goals / Non-Goals - -**Goals:** - - -**Non-Goals:** - - -## DimOS Architecture - - - -## Decisions - - - -## Safety / Simulation / Replay - - - -## Risks / Trade-offs - - - -## Migration / Rollout - - - -## Open Questions - - diff --git a/openspec/schemas/dimos-capability/templates/docs.md b/openspec/schemas/dimos-capability/templates/docs.md deleted file mode 100644 index d274aed653..0000000000 --- a/openspec/schemas/dimos-capability/templates/docs.md +++ /dev/null @@ -1,19 +0,0 @@ -## User-Facing Docs - - - -## Contributor Docs - - - -## Coding-Agent Docs - - - -## Doc Validation - - - -## No Docs Needed - - diff --git a/openspec/schemas/dimos-capability/templates/proposal.md b/openspec/schemas/dimos-capability/templates/proposal.md deleted file mode 100644 index 98d409e8de..0000000000 --- a/openspec/schemas/dimos-capability/templates/proposal.md +++ /dev/null @@ -1,32 +0,0 @@ -## Why - - - -## What Changes - - - -## Affected DimOS Surfaces - - -- Modules/streams: -- Blueprints/CLI: -- Skills/MCP: -- Hardware/simulation/replay: -- Docs/generated registries: - -## Capabilities - -### New Capabilities - -- ``: - -### Modified Capabilities - -- ``: - -## Impact - - diff --git a/openspec/schemas/dimos-capability/templates/spec.md b/openspec/schemas/dimos-capability/templates/spec.md deleted file mode 100644 index afc0c1ff58..0000000000 --- a/openspec/schemas/dimos-capability/templates/spec.md +++ /dev/null @@ -1,16 +0,0 @@ -## ADDED Requirements - -### Requirement: - - -#### Scenario: -- **GIVEN** -- **WHEN** -- **THEN** -- **AND** - - diff --git a/openspec/schemas/dimos-capability/templates/tasks.md b/openspec/schemas/dimos-capability/templates/tasks.md deleted file mode 100644 index b38fcdfabb..0000000000 --- a/openspec/schemas/dimos-capability/templates/tasks.md +++ /dev/null @@ -1,15 +0,0 @@ -## 1. Implementation - -- [ ] 1.1 -- [ ] 1.2 - -## 2. Documentation - -- [ ] 2.1 - -## 3. Verification - -- [ ] 3.1 Run `openspec validate ` -- [ ] 3.2 Run focused tests for changed code -- [ ] 3.3 Run docs validation commands for changed docs -- [ ] 3.4 Manually QA through the relevant DimOS surface (CLI, MCP, simulation/replay, hardware procedure, HTTP API, or library driver) From b348d1c82c4e2454946bd790639ce713e6a9c299 Mon Sep 17 00:00:00 2001 From: cc Date: Tue, 16 Jun 2026 13:22:52 -0700 Subject: [PATCH 10/15] fix: invalidate rendered robot asset cache --- dimos/robot/assets/processing.py | 47 +++++++++++++++++++++++++-- dimos/robot/assets/test_processing.py | 41 +++++++++++++++++++++++ 2 files changed, 85 insertions(+), 3 deletions(-) diff --git a/dimos/robot/assets/processing.py b/dimos/robot/assets/processing.py index cb0ff942fc..d82efa914c 100644 --- a/dimos/robot/assets/processing.py +++ b/dimos/robot/assets/processing.py @@ -61,9 +61,10 @@ def render_urdf( resolved_xacro_args, package_uri_mode, ) - cache_path = _RENDERED_URDF_CACHE_ROOT / cache_key / source_path.stem + rendered_stem = _rendered_urdf_stem(source_path) + cache_path = _RENDERED_URDF_CACHE_ROOT / cache_key / rendered_stem cache_path.mkdir(parents=True, exist_ok=True) - rendered_urdf = cache_path / f"{source_path.stem}.urdf" + rendered_urdf = cache_path / f"{rendered_stem}.urdf" if rendered_urdf.exists(): logger.debug(f"Using cached rendered URDF: {rendered_urdf}") @@ -121,7 +122,7 @@ def _generate_render_key( xacro_args: Mapping[str, str], package_uri_mode: PackageUriMode, ) -> str: - processing_version = "urdf-render-v1" + processing_version = "urdf-render-v2" mtime = urdf_path.stat().st_mtime if urdf_path.exists() else 0 key_data = repr( ( @@ -129,6 +130,7 @@ def _generate_render_key( str(urdf_path), mtime, sorted((name, str(path)) for name, path in package_paths.items()), + _render_dependency_fingerprints(urdf_path, package_paths), sorted(xacro_args.items()), package_uri_mode, ) @@ -136,6 +138,45 @@ def _generate_render_key( return hashlib.sha256(key_data.encode()).hexdigest()[:16] +def _rendered_urdf_stem(source_path: Path) -> str: + """Return a stable output stem without duplicated URDF suffixes.""" + return source_path.stem.removesuffix(".urdf") + + +def _render_dependency_fingerprints( + urdf_path: Path, + package_paths: Mapping[str, Path], +) -> tuple[tuple[str, str], ...]: + """Fingerprint files that can affect Xacro expansion or URI rewriting.""" + roots = list(package_paths.values()) or [urdf_path.parent] + fingerprints = [] + for root in sorted(set(roots), key=str): + fingerprints.append((str(root), _directory_fingerprint(root))) + return tuple(fingerprints) + + +def _directory_fingerprint(root: Path) -> str: + digest = hashlib.sha256() + if not root.exists(): + digest.update(b"missing") + return digest.hexdigest()[:16] + + for path in sorted(root.rglob("*")): + if ".git" in path.parts: + continue + try: + stat = path.stat() + except OSError: + continue + if not path.is_file(): + continue + relative_path = path.relative_to(root) + digest.update(str(relative_path).encode()) + digest.update(str(stat.st_size).encode()) + digest.update(str(stat.st_mtime_ns).encode()) + return digest.hexdigest()[:16] + + def _process_xacro( xacro_path: Path, package_paths: dict[str, Path], diff --git a/dimos/robot/assets/test_processing.py b/dimos/robot/assets/test_processing.py index 798dcd556e..78984de50d 100644 --- a/dimos/robot/assets/test_processing.py +++ b/dimos/robot/assets/test_processing.py @@ -60,3 +60,44 @@ def test_render_urdf_can_rewrite_package_uris_to_absolute_paths( rendered_text = rendered.read_text() assert "package://" not in rendered_text assert str(mesh) in rendered_text + + +def test_render_urdf_cache_key_tracks_package_root_changes( + tmp_path: Path, + monkeypatch, +) -> None: + monkeypatch.setattr(processing, "_RENDERED_URDF_CACHE_ROOT", tmp_path / "rendered") + package_root = tmp_path / "pkg" + mesh = package_root / "meshes" / "link.stl" + mesh.parent.mkdir(parents=True) + mesh.write_text("solid old\nendsolid old\n") + urdf = tmp_path / "robot.urdf" + urdf.write_text( + "" + "" + "" + ) + + first = processing.render_urdf(urdf, {"pkg": package_root}) + mesh.write_text("solid new\nendsolid new\n") + second = processing.render_urdf(urdf, {"pkg": package_root}) + + assert second != first + + +def test_render_urdf_strips_nested_urdf_suffix_from_cache_name( + tmp_path: Path, + monkeypatch, +) -> None: + monkeypatch.setattr(processing, "_RENDERED_URDF_CACHE_ROOT", tmp_path / "rendered") + xacro = tmp_path / "robot.urdf.xacro" + xacro.write_text("") + monkeypatch.setattr( + processing, + "_process_xacro", + lambda _path, _package_paths, _xacro_args: "", + ) + + rendered = processing.render_urdf(xacro) + + assert rendered.name == "robot.urdf" From a22ca9836c2c443598fe34ea15d01d3379f1946f Mon Sep 17 00:00:00 2001 From: cc Date: Fri, 19 Jun 2026 18:27:46 -0700 Subject: [PATCH 11/15] test: revise unit test --- dimos/robot/assets/processing.py | 2 +- dimos/robot/assets/test_manager.py | 94 +++++++++++-------- .../0001-git-backed-robot-asset-manager.md | 7 -- 3 files changed, 55 insertions(+), 48 deletions(-) delete mode 100644 docs/adr/0001-git-backed-robot-asset-manager.md diff --git a/dimos/robot/assets/processing.py b/dimos/robot/assets/processing.py index d82efa914c..4a2907a69f 100644 --- a/dimos/robot/assets/processing.py +++ b/dimos/robot/assets/processing.py @@ -173,7 +173,7 @@ def _directory_fingerprint(root: Path) -> str: relative_path = path.relative_to(root) digest.update(str(relative_path).encode()) digest.update(str(stat.st_size).encode()) - digest.update(str(stat.st_mtime_ns).encode()) + digest.update(path.read_bytes()) return digest.hexdigest()[:16] diff --git a/dimos/robot/assets/test_manager.py b/dimos/robot/assets/test_manager.py index a231ba750a..a8751e49df 100644 --- a/dimos/robot/assets/test_manager.py +++ b/dimos/robot/assets/test_manager.py @@ -14,12 +14,11 @@ import os from pathlib import Path -import subprocess import pytest from dimos.robot.assets.declarations import ROBOT_ASSETS -from dimos.robot.assets.git_cache import GitAssetCache +from dimos.robot.assets.git_cache import GitAssetCache, GitAssetCheckout from dimos.robot.assets.manager import ( ArtifactRole, RobotAssetDeclaration, @@ -33,46 +32,46 @@ ) -def _git(cwd: Path, *args: str) -> str: - return subprocess.run( - ["git", *args], - cwd=cwd, - check=True, - capture_output=True, - text=True, - ).stdout.strip() +class RecordingGitAssetCache(GitAssetCache): + def __init__(self, checkout_path: Path) -> None: + self.checkout_path = checkout_path + self.resolve_calls: list[tuple[str, str]] = [] + + def resolve(self, repo_url: str, ref: str) -> GitAssetCheckout: + self.resolve_calls.append((repo_url, ref)) + return GitAssetCheckout(path=self.checkout_path, repo_url=repo_url, ref=ref) @pytest.fixture() -def asset_manager(tmp_path: Path) -> RobotAssetManager: - repo = tmp_path / "robot_assets" - repo.mkdir() - _git(repo, "init", "-b", "main") - _git(repo, "config", "user.email", "test@example.com") - _git(repo, "config", "user.name", "Test User") - (repo / "robots" / "testbot").mkdir(parents=True) - (repo / "robots" / "testbot" / "model.urdf").write_text("") - (repo / "packages" / "testbot_description").mkdir(parents=True) - (repo / "packages" / "testbot_description" / "package.xml").write_text("") - _git(repo, "add", ".") - _git(repo, "commit", "-m", "assets") +def asset_manager(tmp_path: Path) -> tuple[RobotAssetManager, RecordingGitAssetCache]: + checkout = tmp_path / "checkout" + (checkout / "robots" / "testbot").mkdir(parents=True) + (checkout / "robots" / "testbot" / "model.urdf").write_text("") + (checkout / "packages" / "testbot_description").mkdir(parents=True) + (checkout / "packages" / "testbot_description" / "package.xml").write_text("") declaration = RobotAssetDeclaration( model="testbot", - repo_url=str(repo), + repo_url="https://example.invalid/testbot.git", ref="main", artifacts={"urdf": "robots/testbot/model.urdf"}, package_roots={"testbot_description": "packages/testbot_description"}, ) - return RobotAssetManager( + git_cache = RecordingGitAssetCache(checkout) + manager = RobotAssetManager( {"testbot": declaration}, - git_cache=GitAssetCache(tmp_path / "cache"), + git_cache=git_cache, ) + return manager, git_cache + +def test_resolves_artifact_paths_and_package_roots( + asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], +) -> None: + manager, _git_cache = asset_manager -def test_resolves_artifact_paths_and_package_roots(asset_manager: RobotAssetManager) -> None: - artifact = asset_manager.resolve_artifact("testbot", ArtifactRole.URDF) - package_root = asset_manager.resolve_package_root("testbot", "testbot_description") + artifact = manager.resolve_artifact("testbot", ArtifactRole.URDF) + package_root = manager.resolve_package_root("testbot", "testbot_description") assert artifact.name == "model.urdf" assert artifact.read_text() == "" @@ -80,34 +79,48 @@ def test_resolves_artifact_paths_and_package_roots(asset_manager: RobotAssetMana assert (package_root / "package.xml").exists() -def test_unknown_model_and_undeclared_artifact_role_raise(asset_manager: RobotAssetManager) -> None: +def test_unknown_model_and_undeclared_artifact_role_raise( + asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], +) -> None: + manager, _git_cache = asset_manager + with pytest.raises(RobotAssetError, match="Unknown robot asset model 'missing'"): - asset_manager.resolve_artifact("missing", ArtifactRole.URDF) + manager.resolve_artifact("missing", ArtifactRole.URDF) with pytest.raises(RobotAssetError, match="does not declare artifact role 'mjcf'"): - asset_manager.resolve_artifact("testbot", ArtifactRole.MJCF) + manager.resolve_artifact("testbot", ArtifactRole.MJCF) -def test_lazy_asset_paths_resolve_only_on_path_operations(asset_manager: RobotAssetManager) -> None: - artifact_path = RobotAssetPath("testbot", ArtifactRole.URDF, manager=asset_manager) - package_path = RobotAssetPackagePath("testbot", "testbot_description", manager=asset_manager) +def test_lazy_asset_paths_defer_checkout_until_path_operations( + asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], +) -> None: + manager, git_cache = asset_manager - assert object.__getattribute__(artifact_path, "_robot_asset_resolved_cache") is None - assert object.__getattribute__(package_path, "_robot_asset_resolved_cache") is None + artifact_path = RobotAssetPath("testbot", ArtifactRole.URDF, manager=manager) + package_path = RobotAssetPackagePath("testbot", "testbot_description", manager=manager) + + assert git_cache.resolve_calls == [] artifact_string = str(artifact_path) assert artifact_string.endswith("robots/testbot/model.urdf") - assert object.__getattribute__(artifact_path, "_robot_asset_resolved_cache") is not None + assert git_cache.resolve_calls == [("https://example.invalid/testbot.git", "main")] assert os.fspath(package_path).endswith("packages/testbot_description") - assert object.__getattribute__(package_path, "_robot_asset_resolved_cache") is not None + assert git_cache.resolve_calls == [ + ("https://example.invalid/testbot.git", "main"), + ("https://example.invalid/testbot.git", "main"), + ] assert artifact_path.exists() assert (package_path / "package.xml").exists() -def test_default_manager_can_be_injected(asset_manager: RobotAssetManager) -> None: - set_default_robot_asset_manager(asset_manager) +def test_default_manager_can_be_injected( + asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], +) -> None: + manager, _git_cache = asset_manager + + set_default_robot_asset_manager(manager) try: assert robot_asset_package_paths("testbot")["testbot_description"].exists() assert robot_asset_xacro_args("testbot") == {} @@ -116,6 +129,7 @@ def test_default_manager_can_be_injected(asset_manager: RobotAssetManager) -> No def test_robot_asset_declarations_are_static_and_consistent() -> None: + assert set(ROBOT_ASSETS) == {"a750", "piper", "xarm6", "xarm7"} known_roles = {role.value for role in ArtifactRole} | {"urdf_ik"} for key, declaration in ROBOT_ASSETS.items(): diff --git a/docs/adr/0001-git-backed-robot-asset-manager.md b/docs/adr/0001-git-backed-robot-asset-manager.md deleted file mode 100644 index b4cb9ab32b..0000000000 --- a/docs/adr/0001-git-backed-robot-asset-manager.md +++ /dev/null @@ -1,7 +0,0 @@ -# Use a Git-backed Robot Asset Manager for robot model files - -DimOS will replace selected Git LFS robot description bundles with a Git-backed Robot Asset Manager that resolves robot model files from upstream robot description sources into a standard user cache. The design prioritizes ease of use and avoiding copied asset bundles in the DimOS repo: model assets are declared robot-first with typed Python objects, use branch/tag/commit refs, deduplicate checkouts by source, update clean cached repos when upstream changes are available, warn and continue with cache on update failure, and skip updates when local cache changes are present. - -We will build this as a thin DimOS layer over existing Git tooling rather than writing Git operations from scratch or depending on `robot_descriptions.py` as the primary abstraction. The robot asset layer exposes flat artifact keys such as `urdf`, `mjcf`, `srdf`, `mesh_dir`, and additional string keys when needed; Xacro files remain ordinary resolved artifacts and are processed by the existing model parsing and Drake preparation layers using declared ROS package roots and xacro arguments. - -This accepts less strict reproducibility by default than commit-only pinning, but keeps commit refs available for cases that need them while making the common path simple and fresh against upstream robot description sources. From c91022214cd7b06b45b0914b054624551fa43789 Mon Sep 17 00:00:00 2001 From: cc Date: Fri, 19 Jun 2026 19:19:57 -0700 Subject: [PATCH 12/15] refactor: simplify robot asset sources --- CONTEXT.md | 14 +- .../manipulation/test_manipulation_module.py | 8 +- dimos/robot/assets/README.md | 49 ++- dimos/robot/assets/declarations.py | 79 ----- dimos/robot/assets/git_cache.py | 6 +- dimos/robot/assets/manager.py | 294 ------------------ dimos/robot/assets/source.py | 154 +++++++++ dimos/robot/assets/test_manager.py | 149 --------- dimos/robot/assets/test_source.py | 117 +++++++ dimos/robot/catalog/a750.py | 10 +- dimos/robot/catalog/piper.py | 17 +- dimos/robot/catalog/ufactory.py | 27 +- docs/capabilities/manipulation/a750.md | 4 +- .../manipulation/adding_a_custom_arm.md | 15 +- docs/capabilities/manipulation/readme.md | 6 +- docs/coding-agents/code-quality-rules.md | 2 +- pyproject.toml | 1 + uv.lock | 2 + 18 files changed, 360 insertions(+), 594 deletions(-) delete mode 100644 dimos/robot/assets/declarations.py delete mode 100644 dimos/robot/assets/manager.py create mode 100644 dimos/robot/assets/source.py delete mode 100644 dimos/robot/assets/test_manager.py create mode 100644 dimos/robot/assets/test_source.py diff --git a/CONTEXT.md b/CONTEXT.md index 58d241d682..cdab58e9bb 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -16,18 +16,18 @@ _Avoid_: LFS replacement, description downloader, asset loader The standard user cache directory where DimOS stores fetched robot description source checkouts for reuse across runs. _Avoid_: Data directory, vendored assets, repository assets -**Robot Asset Manifest**: -A DimOS-maintained declaration of robot model assets, their robot description sources, default revisions, package roots, and provenance metadata. It may be represented as typed Python objects; it does not imply a YAML/TOML file. -_Avoid_: Registry, asset list, dependency file, YAML manifest +**Robot Asset Supply Path**: +The source and resolution route by which DimOS obtains robot model assets for both built-in and user-supplied robots. +_Avoid_: Built-in asset path, custom asset path, deployment path + +**Robot Description Source Handle**: +A path-like object that names a robot description source and resolves to the local checkout root for that source. Callers form concrete model paths by joining relative paths from this root. +_Avoid_: Registry entry, artifact lookup, asset manifest **ROS Package Root**: The local directory corresponding to a ROS-style package name, used to resolve `package://...` URIs and `$(find package_name)` expressions in robot model files. _Avoid_: Package path, asset package, Python package -**Artifact Role**: -A string key naming a supported robot model asset file or directory kind. Common roles include `urdf`, `mjcf`, `srdf`, and `mesh_dir`; extra role keys such as `urdf_ik` may be used when a robot needs additional files. Strings are the canonical internal representation. -_Avoid_: Parser mode, arbitrary attachment, file purpose - **DimOS Robot Model Config**: A DimOS configuration object that names the model paths, package paths, joints, links, and robot-specific metadata needed by planning, control, simulation, or visualization. _Avoid_: Robot description, URDF config diff --git a/dimos/manipulation/test_manipulation_module.py b/dimos/manipulation/test_manipulation_module.py index 471453bfcf..f666443cac 100644 --- a/dimos/manipulation/test_manipulation_module.py +++ b/dimos/manipulation/test_manipulation_module.py @@ -36,7 +36,7 @@ from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 from dimos.msgs.sensor_msgs.JointState import JointState -from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.catalog.ufactory import _XARM_MODEL_PATH, _XARM_PACKAGE_PATHS pytestmark = pytest.mark.self_hosted @@ -47,7 +47,7 @@ def _drake_available() -> bool: def _xarm_urdf_available() -> bool: try: - model_path = RobotAssetPath("xarm7", "urdf") + model_path = _XARM_MODEL_PATH return model_path.exists() except Exception: return False @@ -57,12 +57,12 @@ def _get_xarm7_config() -> RobotModelConfig: """Create XArm7 robot config for testing.""" return RobotModelConfig( name="test_arm", - model_path=RobotAssetPath("xarm7", "urdf"), + model_path=_XARM_MODEL_PATH, base_pose=PoseStamped(position=Vector3(), orientation=Quaternion()), joint_names=["joint1", "joint2", "joint3", "joint4", "joint5", "joint6", "joint7"], end_effector_link="link7", base_link="link_base", - package_paths=robot_asset_package_paths("xarm7"), + package_paths=_XARM_PACKAGE_PATHS, xacro_args={"dof": "7", "limited": "true"}, auto_convert_meshes=True, max_velocity=1.0, diff --git a/dimos/robot/assets/README.md b/dimos/robot/assets/README.md index 1dfbb9e0a8..b6b552e5af 100644 --- a/dimos/robot/assets/README.md +++ b/dimos/robot/assets/README.md @@ -9,7 +9,7 @@ not add compatibility wrappers outside this module for new code. Import directly from the source modules, for example: ```python -from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.source import RobotDescriptionSource ``` There is no `__init__.py` on purpose: DimOS disallows package `__init__.py` files @@ -20,7 +20,7 @@ except at the root package to avoid accidental import side effects. Assets live under: ```text -~/.cache/dimos/robot_assets/ +/dimos/robot_assets/ ├── sources/ # Git checkouts by source identity ├── locks/ # per-source file locks └── derived/ @@ -35,46 +35,41 @@ Assets live under: - warn and keep cached content if update fails; - warn and skip update for dirty cached repos, preserving local edits. -## Declaring a robot asset +## Using a robot description source -Add declarations in `declarations.py`: +Create a source handle wherever the robot adapter or catalog is defined, then +join paths from the repository root: ```python -from dimos.robot.assets.manager import RobotAssetDeclaration - -ROBOT_ASSETS["myarm"] = RobotAssetDeclaration( - model="myarm", - repo_url="https://github.com/example/myarm_description", - ref="main", # branch, tag, or commit - artifacts={ - "urdf": "urdf/myarm.urdf.xacro", - "mesh_dir": "meshes", - }, - package_roots={"myarm_description": "."}, - xacro_args={"limited": "true"}, +from dimos.robot.assets.source import RobotDescriptionSource + +_MYARM_REPO = RobotDescriptionSource( + url="https://github.com/example/myarm_description", + ref="main", ) -``` -Artifact role keys are strings. Common roles are `urdf`, `mjcf`, `srdf`, and -`mesh_dir`; extra flat roles such as `urdf_ik` are allowed when a catalog needs -an additional model variant. +model_path = _MYARM_REPO / "urdf" / "myarm.urdf.xacro" +package_paths = {"myarm_description": _MYARM_REPO / "."} +``` -`package_roots` maps ROS package names to directories inside the checkout. These -roots are used for `package://...` URIs and Xacro `$(find package_name)`. +Package roots map ROS package names to source-relative directories. These roots +are used for `package://...` URIs and Xacro `$(find package_name)`. ## Using assets in catalogs Catalogs should stay lazy at import time: ```python -from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.source import RobotDescriptionSource + +_MYARM_REPO = RobotDescriptionSource(url="https://github.com/example/myarm_description", ref="main") -model_path = RobotAssetPath("myarm", "urdf") -package_paths = robot_asset_package_paths("myarm") +model_path = _MYARM_REPO / "urdf" / "myarm.urdf.xacro" +package_paths = {"myarm_description": _MYARM_REPO / "."} ``` -`RobotAssetPath` and `RobotAssetPackagePath` defer clone/update/path validation -until path operations such as `str(path)`, `path.resolve()`, or `path.exists()`. +`RobotDescriptionPath` defers clone/update/path validation until path operations +such as `str(path)`, `path.resolve()`, or `path.exists()`. ## Rendering URDFs diff --git a/dimos/robot/assets/declarations.py b/dimos/robot/assets/declarations.py deleted file mode 100644 index a41fbe5ae4..0000000000 --- a/dimos/robot/assets/declarations.py +++ /dev/null @@ -1,79 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Robot asset declarations resolved by :mod:`dimos.robot.assets`.""" - -from __future__ import annotations - -from dimos.robot.assets.manager import RobotAssetDeclaration - -XARM_ROS2_REPO = "https://github.com/xArm-Developer/xarm_ros2" -PIPER_DESCRIPTION_REPO = "https://github.com/agilexrobotics/agx_arm_urdf" -A750_DESCRIPTION_REPO = "https://github.com/adob/a750_description" - - -ROBOT_ASSETS: dict[str, RobotAssetDeclaration] = { - "xarm6": RobotAssetDeclaration( - model="xarm6", - repo_url=XARM_ROS2_REPO, - ref="humble", - artifacts={ - "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", - "mesh_dir": "xarm_description/meshes", - }, - package_roots={"xarm_description": "xarm_description"}, - xacro_args={"dof": "6", "limited": "true"}, - source_name="xarm_ros2", - ), - "xarm7": RobotAssetDeclaration( - model="xarm7", - repo_url=XARM_ROS2_REPO, - ref="humble", - artifacts={ - "urdf": "xarm_description/urdf/xarm_device.urdf.xacro", - "mesh_dir": "xarm_description/meshes", - }, - package_roots={"xarm_description": "xarm_description"}, - xacro_args={"dof": "7", "limited": "true"}, - source_name="xarm_ros2", - ), - "piper": RobotAssetDeclaration( - model="piper", - repo_url=PIPER_DESCRIPTION_REPO, - ref="main", - artifacts={ - "urdf": "piper/urdf/piper_with_gripper_description.xacro", - "urdf_ik": "piper/urdf/piper_description.urdf", - "mesh_dir": "piper/meshes", - }, - # Upstream URDFs reference package://agx_arm_description/agx_arm_urdf/... - # and expect the checkout directory to be named agx_arm_urdf inside the - # package root. GitAssetCache preserves that checkout directory name. - package_roots={"agx_arm_description": ".."}, - source_name="agx_arm_urdf", - license="MIT", - ), - "a750": RobotAssetDeclaration( - model="a750", - repo_url=A750_DESCRIPTION_REPO, - ref="master", - artifacts={ - "urdf": "urdf/a750_rev1.urdf", - "mesh_dir": "meshes/a750_rev1", - }, - package_roots={"a750_description": "."}, - source_name="a750_description", - license="MIT", - ), -} diff --git a/dimos/robot/assets/git_cache.py b/dimos/robot/assets/git_cache.py index 4c6b9f7149..1554cf5476 100644 --- a/dimos/robot/assets/git_cache.py +++ b/dimos/robot/assets/git_cache.py @@ -26,9 +26,9 @@ from filelock import FileLock from git import GitCommandError, InvalidGitRepositoryError, NoSuchPathError, Repo +from platformdirs import user_cache_path -DEFAULT_GIT_ASSET_CACHE_ROOT = Path.home() / ".cache" / "dimos" / "robot_assets" -DEFAULT_ROBOT_ASSET_CACHE_ROOT = DEFAULT_GIT_ASSET_CACHE_ROOT +DEFAULT_ROBOT_ASSET_CACHE_ROOT = user_cache_path("dimos", appauthor=False) / "robot_assets" class GitAssetCacheError(RuntimeError): @@ -61,7 +61,7 @@ class GitAssetCache: - if the cached repository has local changes, warn and skip updates. """ - def __init__(self, cache_root: Path | str = DEFAULT_GIT_ASSET_CACHE_ROOT) -> None: + def __init__(self, cache_root: Path | str = DEFAULT_ROBOT_ASSET_CACHE_ROOT) -> None: self.cache_root = Path(cache_root).expanduser() self._sources_root = self.cache_root / "sources" self._locks_root = self.cache_root / "locks" diff --git a/dimos/robot/assets/manager.py b/dimos/robot/assets/manager.py deleted file mode 100644 index 3195a91639..0000000000 --- a/dimos/robot/assets/manager.py +++ /dev/null @@ -1,294 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -"""Robot model asset declarations and lazy path adapters.""" - -from __future__ import annotations - -from collections.abc import Mapping -from dataclasses import dataclass, field -from enum import Enum -from pathlib import Path - -from dimos.robot.assets.git_cache import GitAssetCache - - -class RobotAssetError(RuntimeError): - """Raised when a robot asset declaration cannot satisfy a request.""" - - -class ArtifactRole(str, Enum): - """Common robot asset artifact roles. - - Strings are canonical internally; this enum is a convenience for common roles. - """ - - URDF = "urdf" - MJCF = "mjcf" - SRDF = "srdf" - MESH_DIR = "mesh_dir" - - -@dataclass(frozen=True) -class RobotAssetDeclaration: - """Typed declaration for one robot model's assets.""" - - model: str - repo_url: str - ref: str - artifacts: Mapping[str, str] - package_roots: Mapping[str, str] = field(default_factory=dict) - xacro_args: Mapping[str, str] = field(default_factory=dict) - source_name: str | None = None - license: str | None = None - - -class RobotAssetManager: - """Resolve robot model artifacts and package roots from declarations.""" - - def __init__( - self, - declarations: Mapping[str, RobotAssetDeclaration] | None = None, - git_cache: GitAssetCache | None = None, - ) -> None: - self._declarations = dict(declarations or {}) - self._git_cache = git_cache or GitAssetCache() - - def get_declaration(self, model: str) -> RobotAssetDeclaration: - try: - return self._declarations[model] - except KeyError as exc: - available = ", ".join(sorted(self._declarations)) or "none" - raise RobotAssetError( - f"Unknown robot asset model {model!r}. Available models: {available}." - ) from exc - - def resolve_artifact(self, model: str, role: str | ArtifactRole) -> Path: - declaration = self.get_declaration(model) - role_key = _role_key(role) - try: - relative_path = declaration.artifacts[role_key] - except KeyError as exc: - available = ", ".join(sorted(declaration.artifacts)) or "none" - raise RobotAssetError( - f"Robot asset model {model!r} does not declare artifact role {role_key!r}. " - f"Available roles: {available}." - ) from exc - - path = self._checkout(declaration) / relative_path - if not path.exists(): - raise RobotAssetError( - f"Declared artifact {role_key!r} for robot asset model {model!r} does not exist: {path}" - ) - return path - - def resolve_package_root(self, model: str, package_name: str) -> Path: - declaration = self.get_declaration(model) - try: - relative_path = declaration.package_roots[package_name] - except KeyError as exc: - available = ", ".join(sorted(declaration.package_roots)) or "none" - raise RobotAssetError( - f"Robot asset model {model!r} does not declare ROS package root " - f"{package_name!r}. Available package roots: {available}." - ) from exc - - path = self._checkout(declaration) / relative_path - if not path.exists(): - raise RobotAssetError( - f"Declared package root {package_name!r} for robot asset model {model!r} " - f"does not exist: {path}" - ) - return path - - def package_roots(self, model: str) -> dict[str, Path]: - declaration = self.get_declaration(model) - return { - package_name: RobotAssetPackagePath(model, package_name, manager=self) - for package_name in declaration.package_roots - } - - def xacro_args(self, model: str) -> dict[str, str]: - return dict(self.get_declaration(model).xacro_args) - - def _checkout(self, declaration: RobotAssetDeclaration) -> Path: - return self._git_cache.resolve(declaration.repo_url, declaration.ref).path - - -class RobotAssetPath(type(Path())): # type: ignore[misc] - """Lazy Path-like adapter for a declared robot model artifact.""" - - def __new__( - cls, - model: str, - role: str | ArtifactRole, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> RobotAssetPath: - instance: RobotAssetPath = super().__new__(cls, ".") - object.__setattr__(instance, "_robot_asset_model", model) - object.__setattr__(instance, "_robot_asset_role", _role_key(role)) - object.__setattr__( - instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) - ) - object.__setattr__( - instance, "_robot_asset_manager", manager or default_robot_asset_manager() - ) - object.__setattr__(instance, "_robot_asset_resolved_cache", None) - return instance - - def __init__( - self, - model: str, - role: str | ArtifactRole, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> None: - del model, role, relative_parts, manager - - def _resolve(self) -> Path: - cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") - if cache is None: - manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") - model = object.__getattribute__(self, "_robot_asset_model") - role = object.__getattribute__(self, "_robot_asset_role") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - cache = manager.resolve_artifact(model, role).joinpath(*relative_parts) - object.__setattr__(self, "_robot_asset_resolved_cache", cache) - return cache - - def __getattribute__(self, name: str) -> object: - try: - object.__getattribute__(self, "_robot_asset_model") - except AttributeError: - return object.__getattribute__(self, name) - - if name.startswith("_robot_asset_") or name in {"_resolve"}: - return object.__getattribute__(self, name) - - return getattr(object.__getattribute__(self, "_resolve")(), name) - - def __str__(self) -> str: - return str(self._resolve()) - - def __fspath__(self) -> str: - return str(self._resolve()) - - def __truediv__(self, other: object) -> RobotAssetPath: - model = object.__getattribute__(self, "_robot_asset_model") - role = object.__getattribute__(self, "_robot_asset_role") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - manager = object.__getattribute__(self, "_robot_asset_manager") - return RobotAssetPath(model, role, *relative_parts, other, manager=manager) - - -class RobotAssetPackagePath(type(Path())): # type: ignore[misc] - """Lazy Path-like adapter for a declared ROS package root.""" - - def __new__( - cls, - model: str, - package_name: str, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> RobotAssetPackagePath: - instance: RobotAssetPackagePath = super().__new__(cls, ".") - object.__setattr__(instance, "_robot_asset_model", model) - object.__setattr__(instance, "_robot_asset_package_name", package_name) - object.__setattr__( - instance, "_robot_asset_relative_parts", tuple(str(p) for p in relative_parts) - ) - object.__setattr__( - instance, "_robot_asset_manager", manager or default_robot_asset_manager() - ) - object.__setattr__(instance, "_robot_asset_resolved_cache", None) - return instance - - def __init__( - self, - model: str, - package_name: str, - *relative_parts: object, - manager: RobotAssetManager | None = None, - ) -> None: - del model, package_name, relative_parts, manager - - def _resolve(self) -> Path: - cache: Path | None = object.__getattribute__(self, "_robot_asset_resolved_cache") - if cache is None: - manager: RobotAssetManager = object.__getattribute__(self, "_robot_asset_manager") - model = object.__getattribute__(self, "_robot_asset_model") - package_name = object.__getattribute__(self, "_robot_asset_package_name") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - cache = manager.resolve_package_root(model, package_name).joinpath(*relative_parts) - object.__setattr__(self, "_robot_asset_resolved_cache", cache) - return cache - - def __getattribute__(self, name: str) -> object: - try: - object.__getattribute__(self, "_robot_asset_model") - except AttributeError: - return object.__getattribute__(self, name) - - if name.startswith("_robot_asset_") or name in {"_resolve"}: - return object.__getattribute__(self, name) - - return getattr(object.__getattribute__(self, "_resolve")(), name) - - def __str__(self) -> str: - return str(self._resolve()) - - def __fspath__(self) -> str: - return str(self._resolve()) - - def __truediv__(self, other: object) -> RobotAssetPackagePath: - model = object.__getattribute__(self, "_robot_asset_model") - package_name = object.__getattribute__(self, "_robot_asset_package_name") - relative_parts = object.__getattribute__(self, "_robot_asset_relative_parts") - manager = object.__getattribute__(self, "_robot_asset_manager") - return RobotAssetPackagePath(model, package_name, *relative_parts, other, manager=manager) - - -_DEFAULT_MANAGER: RobotAssetManager | None = None - - -def default_robot_asset_manager() -> RobotAssetManager: - global _DEFAULT_MANAGER - if _DEFAULT_MANAGER is None: - from dimos.robot.assets.declarations import ROBOT_ASSETS - - _DEFAULT_MANAGER = RobotAssetManager(ROBOT_ASSETS) - return _DEFAULT_MANAGER - - -def set_default_robot_asset_manager(manager: RobotAssetManager | None) -> None: - """Override the process-default robot asset manager. - - Passing ``None`` clears the override and restores lazy construction from the - DimOS declarations module on the next default lookup. - """ - global _DEFAULT_MANAGER - _DEFAULT_MANAGER = manager - - -def robot_asset_package_paths(model: str) -> dict[str, Path]: - return default_robot_asset_manager().package_roots(model) - - -def robot_asset_xacro_args(model: str) -> dict[str, str]: - return default_robot_asset_manager().xacro_args(model) - - -def _role_key(role: str | ArtifactRole) -> str: - return role.value if isinstance(role, ArtifactRole) else str(role) diff --git a/dimos/robot/assets/source.py b/dimos/robot/assets/source.py new file mode 100644 index 0000000000..3a93298da6 --- /dev/null +++ b/dimos/robot/assets/source.py @@ -0,0 +1,154 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Lazy path handles for Git-backed robot description sources.""" + +from __future__ import annotations + +from pathlib import Path + +from dimos.robot.assets.git_cache import GitAssetCache + + +class RobotDescriptionSource: + """Git-backed robot description source root. + + Joining paths is lazy. The Git checkout is only resolved when the resulting + :class:`RobotDescriptionPath` is observed as a filesystem path. + """ + + def __init__( + self, + url: str, + ref: str, + git_cache: GitAssetCache | None = None, + ) -> None: + self.url = url + self.ref = ref + self._git_cache = git_cache or GitAssetCache() + self._checkout_path_cache: Path | None = None + + def checkout_path(self) -> Path: + """Return the local checkout root, cloning/updating if needed.""" + if self._checkout_path_cache is None: + self._checkout_path_cache = self._git_cache.resolve(self.url, self.ref).path + return self._checkout_path_cache + + def path(self) -> RobotDescriptionPath: + """Return a lazy path for the checkout root.""" + return RobotDescriptionPath(self, Path(".")) + + @property + def parent(self) -> RobotDescriptionPath: + """Return the checkout root's parent as a lazy path.""" + return RobotDescriptionPath(self, Path("..")) + + def __truediv__(self, other: object) -> RobotDescriptionPath: + return self.path() / other + + def __repr__(self) -> str: + return f"RobotDescriptionSource(url={self.url!r}, ref={self.ref!r})" + + def __eq__(self, other: object) -> bool: + if not isinstance(other, RobotDescriptionSource): + return NotImplemented + return self.url == other.url and self.ref == other.ref + + def __hash__(self) -> int: + return hash((self.url, self.ref)) + + +class RobotDescriptionPath(type(Path())): # type: ignore[misc] + """Lazy Path subclass rooted at a :class:`RobotDescriptionSource`.""" + + def __new__( + cls, + source: RobotDescriptionSource, + relative_path: Path | str, + ) -> RobotDescriptionPath: + instance: RobotDescriptionPath = super().__new__(cls, ".") + object.__setattr__(instance, "_robot_description_source", source) + object.__setattr__(instance, "_robot_description_relative_path", Path(relative_path)) + object.__setattr__(instance, "_robot_description_resolved_cache", None) + return instance + + def __init__( + self, + source: RobotDescriptionSource, + relative_path: Path | str, + ) -> None: + del source, relative_path + + def _resolve(self) -> Path: + cache: Path | None = object.__getattribute__(self, "_robot_description_resolved_cache") + if cache is None: + source: RobotDescriptionSource = object.__getattribute__( + self, "_robot_description_source" + ) + relative_path: Path = object.__getattribute__(self, "_robot_description_relative_path") + cache = source.checkout_path() / relative_path + object.__setattr__(self, "_robot_description_resolved_cache", cache) + return cache + + def __getattribute__(self, name: str) -> object: + try: + object.__getattribute__(self, "_robot_description_source") + except AttributeError: + return object.__getattribute__(self, name) + + if name.startswith("_robot_description_") or name in {"_resolve"}: + return object.__getattribute__(self, name) + + if name == "parent": + source: RobotDescriptionSource = object.__getattribute__( + self, "_robot_description_source" + ) + relative_path: Path = object.__getattribute__(self, "_robot_description_relative_path") + return RobotDescriptionPath(source, relative_path.parent) + + if name in {"name", "stem", "suffix", "parts"}: + relative_path = object.__getattribute__(self, "_robot_description_relative_path") + return getattr(relative_path, name) + + return getattr(object.__getattribute__(self, "_resolve")(), name) + + def __str__(self) -> str: + return str(self._resolve()) + + def __fspath__(self) -> str: + return str(self._resolve()) + + def __truediv__(self, other: object) -> RobotDescriptionPath: + source: RobotDescriptionSource = object.__getattribute__(self, "_robot_description_source") + relative_path: Path = object.__getattribute__(self, "_robot_description_relative_path") + return RobotDescriptionPath(source, relative_path / str(other)) + + def __eq__(self, other: object) -> bool: + if not isinstance(other, RobotDescriptionPath): + return self._resolve() == other + return object.__getattribute__( + self, "_robot_description_source" + ) == object.__getattribute__( + other, "_robot_description_source" + ) and object.__getattribute__( + self, "_robot_description_relative_path" + ) == object.__getattribute__(other, "_robot_description_relative_path") + + def __hash__(self) -> int: + return hash( + ( + object.__getattribute__(self, "_robot_description_source"), + object.__getattribute__(self, "_robot_description_relative_path"), + ) + ) diff --git a/dimos/robot/assets/test_manager.py b/dimos/robot/assets/test_manager.py deleted file mode 100644 index a8751e49df..0000000000 --- a/dimos/robot/assets/test_manager.py +++ /dev/null @@ -1,149 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import os -from pathlib import Path - -import pytest - -from dimos.robot.assets.declarations import ROBOT_ASSETS -from dimos.robot.assets.git_cache import GitAssetCache, GitAssetCheckout -from dimos.robot.assets.manager import ( - ArtifactRole, - RobotAssetDeclaration, - RobotAssetError, - RobotAssetManager, - RobotAssetPackagePath, - RobotAssetPath, - robot_asset_package_paths, - robot_asset_xacro_args, - set_default_robot_asset_manager, -) - - -class RecordingGitAssetCache(GitAssetCache): - def __init__(self, checkout_path: Path) -> None: - self.checkout_path = checkout_path - self.resolve_calls: list[tuple[str, str]] = [] - - def resolve(self, repo_url: str, ref: str) -> GitAssetCheckout: - self.resolve_calls.append((repo_url, ref)) - return GitAssetCheckout(path=self.checkout_path, repo_url=repo_url, ref=ref) - - -@pytest.fixture() -def asset_manager(tmp_path: Path) -> tuple[RobotAssetManager, RecordingGitAssetCache]: - checkout = tmp_path / "checkout" - (checkout / "robots" / "testbot").mkdir(parents=True) - (checkout / "robots" / "testbot" / "model.urdf").write_text("") - (checkout / "packages" / "testbot_description").mkdir(parents=True) - (checkout / "packages" / "testbot_description" / "package.xml").write_text("") - - declaration = RobotAssetDeclaration( - model="testbot", - repo_url="https://example.invalid/testbot.git", - ref="main", - artifacts={"urdf": "robots/testbot/model.urdf"}, - package_roots={"testbot_description": "packages/testbot_description"}, - ) - git_cache = RecordingGitAssetCache(checkout) - manager = RobotAssetManager( - {"testbot": declaration}, - git_cache=git_cache, - ) - return manager, git_cache - - -def test_resolves_artifact_paths_and_package_roots( - asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], -) -> None: - manager, _git_cache = asset_manager - - artifact = manager.resolve_artifact("testbot", ArtifactRole.URDF) - package_root = manager.resolve_package_root("testbot", "testbot_description") - - assert artifact.name == "model.urdf" - assert artifact.read_text() == "" - assert package_root.name == "testbot_description" - assert (package_root / "package.xml").exists() - - -def test_unknown_model_and_undeclared_artifact_role_raise( - asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], -) -> None: - manager, _git_cache = asset_manager - - with pytest.raises(RobotAssetError, match="Unknown robot asset model 'missing'"): - manager.resolve_artifact("missing", ArtifactRole.URDF) - - with pytest.raises(RobotAssetError, match="does not declare artifact role 'mjcf'"): - manager.resolve_artifact("testbot", ArtifactRole.MJCF) - - -def test_lazy_asset_paths_defer_checkout_until_path_operations( - asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], -) -> None: - manager, git_cache = asset_manager - - artifact_path = RobotAssetPath("testbot", ArtifactRole.URDF, manager=manager) - package_path = RobotAssetPackagePath("testbot", "testbot_description", manager=manager) - - assert git_cache.resolve_calls == [] - - artifact_string = str(artifact_path) - assert artifact_string.endswith("robots/testbot/model.urdf") - assert git_cache.resolve_calls == [("https://example.invalid/testbot.git", "main")] - - assert os.fspath(package_path).endswith("packages/testbot_description") - assert git_cache.resolve_calls == [ - ("https://example.invalid/testbot.git", "main"), - ("https://example.invalid/testbot.git", "main"), - ] - - assert artifact_path.exists() - assert (package_path / "package.xml").exists() - - -def test_default_manager_can_be_injected( - asset_manager: tuple[RobotAssetManager, RecordingGitAssetCache], -) -> None: - manager, _git_cache = asset_manager - - set_default_robot_asset_manager(manager) - try: - assert robot_asset_package_paths("testbot")["testbot_description"].exists() - assert robot_asset_xacro_args("testbot") == {} - finally: - set_default_robot_asset_manager(None) - - -def test_robot_asset_declarations_are_static_and_consistent() -> None: - assert set(ROBOT_ASSETS) == {"a750", "piper", "xarm6", "xarm7"} - known_roles = {role.value for role in ArtifactRole} | {"urdf_ik"} - - for key, declaration in ROBOT_ASSETS.items(): - assert key == declaration.model - assert declaration.repo_url - assert declaration.ref - assert declaration.artifacts - - for role, relative_path in declaration.artifacts.items(): - assert role in known_roles - assert relative_path - assert not Path(relative_path).is_absolute() - - for package_name, relative_path in declaration.package_roots.items(): - assert package_name - assert relative_path - assert not Path(relative_path).is_absolute() diff --git a/dimos/robot/assets/test_source.py b/dimos/robot/assets/test_source.py new file mode 100644 index 0000000000..f7e71a368c --- /dev/null +++ b/dimos/robot/assets/test_source.py @@ -0,0 +1,117 @@ +# Copyright 2025-2026 Dimensional Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from pathlib import Path + +import pytest + +from dimos.robot.assets.git_cache import GitAssetCache, GitAssetCheckout +from dimos.robot.assets.source import RobotDescriptionPath, RobotDescriptionSource + + +class RecordingGitAssetCache(GitAssetCache): + def __init__(self, checkout_path: Path) -> None: + self.checkout_path = checkout_path + self.resolve_calls: list[tuple[str, str]] = [] + + def resolve(self, repo_url: str, ref: str) -> GitAssetCheckout: + self.resolve_calls.append((repo_url, ref)) + return GitAssetCheckout(path=self.checkout_path, repo_url=repo_url, ref=ref) + + +@pytest.fixture() +def robot_source(tmp_path: Path) -> tuple[RobotDescriptionSource, RecordingGitAssetCache]: + checkout = tmp_path / "checkout" / "testbot_description" + (checkout / "robots" / "testbot").mkdir(parents=True) + (checkout / "robots" / "testbot" / "model.urdf").write_text("") + (checkout / "packages" / "testbot_description").mkdir(parents=True) + (checkout / "packages" / "testbot_description" / "package.xml").write_text("") + + git_cache = RecordingGitAssetCache(checkout) + source = RobotDescriptionSource( + url="https://example.invalid/testbot_description.git", + ref="main", + git_cache=git_cache, + ) + return source, git_cache + + +def test_source_path_joining_defers_checkout( + robot_source: tuple[RobotDescriptionSource, RecordingGitAssetCache], +) -> None: + source, git_cache = robot_source + + model_path = source / "robots" / "testbot" / "model.urdf" + package_path = source / "packages" / "testbot_description" + + assert isinstance(model_path, Path) + assert isinstance(model_path, RobotDescriptionPath) + assert git_cache.resolve_calls == [] + + assert str(model_path).endswith("robots/testbot/model.urdf") + assert git_cache.resolve_calls == [("https://example.invalid/testbot_description.git", "main")] + + assert os.fspath(package_path).endswith("packages/testbot_description") + assert git_cache.resolve_calls == [("https://example.invalid/testbot_description.git", "main")] + + assert model_path.exists() + assert (package_path / "package.xml").exists() + + +def test_source_parent_can_express_package_root_without_resolving( + robot_source: tuple[RobotDescriptionSource, RecordingGitAssetCache], +) -> None: + source, git_cache = robot_source + + package_root = source.parent + + assert isinstance(package_root, RobotDescriptionPath) + assert git_cache.resolve_calls == [] + assert package_root.resolve().name == "checkout" + assert git_cache.resolve_calls == [("https://example.invalid/testbot_description.git", "main")] + + +def test_lazy_path_metadata_uses_relative_path_without_checkout( + robot_source: tuple[RobotDescriptionSource, RecordingGitAssetCache], +) -> None: + source, git_cache = robot_source + + model_path = source / "robots" / "testbot" / "model.urdf" + + assert model_path.name == "model.urdf" + assert model_path.stem == "model" + assert model_path.suffix == ".urdf" + assert model_path.parent.name == "testbot" + assert git_cache.resolve_calls == [] + + +def test_custom_source_handle_needs_no_registration(tmp_path: Path) -> None: + checkout = tmp_path / "custom_robot_description" + (checkout / "urdf").mkdir(parents=True) + (checkout / "urdf" / "custom.urdf").write_text("") + git_cache = RecordingGitAssetCache(checkout) + + custom_source = RobotDescriptionSource( + url="https://example.invalid/custom_robot_description.git", + ref="feature/custom", + git_cache=git_cache, + ) + + model_path = custom_source / "urdf" / "custom.urdf" + + assert model_path.read_text() == "" + assert git_cache.resolve_calls == [ + ("https://example.invalid/custom_robot_description.git", "feature/custom") + ] diff --git a/dimos/robot/catalog/a750.py b/dimos/robot/catalog/a750.py index 36ff72f6d4..bf66272a95 100644 --- a/dimos/robot/catalog/a750.py +++ b/dimos/robot/catalog/a750.py @@ -19,10 +19,14 @@ import math from typing import Any -from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.source import RobotDescriptionSource from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath +A750_DESCRIPTION_REPO = "https://github.com/adob/a750_description" +_A750_REPO = RobotDescriptionSource(url=A750_DESCRIPTION_REPO, ref="master") +_A750_PACKAGE_PATHS = {"a750_description": _A750_REPO / "."} + # Static no-gripper URDF for Pinocchio FK. The upstream source only publishes the # full gripper model, so this generated FK variant intentionally stays on LFS. A750_FK_MODEL = LfsPath("a750_description/urdf/a750_rev1_no_gripper.urdf") @@ -73,7 +77,7 @@ def a750( """ defaults: dict[str, Any] = { "name": name, - "model_path": RobotAssetPath("a750", "urdf"), + "model_path": _A750_REPO / "urdf" / "a750_rev1.urdf", "end_effector_link": "gripper_base", "adapter_type": adapter_type, "address": device_path, @@ -81,7 +85,7 @@ def a750( "base_link": "base_link", "home_joints": [0.0, 0.0, -math.radians(90), 0.0, 0.0, 0.0], "base_pose": [0, 0, 0, 0, 0, 0, 1], # base_pose is where the robot sits in the world - "package_paths": robot_asset_package_paths("a750"), + "package_paths": _A750_PACKAGE_PATHS, "xacro_args": {}, "auto_convert_meshes": True, "collision_exclusion_pairs": A750_GRIPPER_COLLISION_EXCLUSIONS, diff --git a/dimos/robot/catalog/piper.py b/dimos/robot/catalog/piper.py index 17fd283a76..103430a0d2 100644 --- a/dimos/robot/catalog/piper.py +++ b/dimos/robot/catalog/piper.py @@ -18,12 +18,21 @@ from typing import Any -from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.source import RobotDescriptionSource from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath +PIPER_DESCRIPTION_REPO = "https://github.com/agilexrobotics/agx_arm_urdf" +_PIPER_REPO = RobotDescriptionSource(url=PIPER_DESCRIPTION_REPO, ref="main") +_PIPER_PACKAGE_PATHS = { + # Upstream URDFs reference package://agx_arm_description/agx_arm_urdf/... + # and expect the checkout directory to be named agx_arm_urdf inside the + # package root. GitAssetCache preserves that checkout directory name. + "agx_arm_description": _PIPER_REPO.parent, +} + # Static no-gripper URDF for Pinocchio FK (xacro not supported by Pinocchio) -PIPER_FK_MODEL = RobotAssetPath("piper", "urdf_ik") +PIPER_FK_MODEL = _PIPER_REPO / "piper" / "urdf" / "piper_description.urdf" # Simulation model path (MJCF) PIPER_SIM_PATH = LfsPath("piper/scene.xml") @@ -60,7 +69,7 @@ def piper( """ defaults: dict[str, Any] = { "name": name, - "model_path": RobotAssetPath("piper", "urdf"), + "model_path": _PIPER_REPO / "piper" / "urdf" / "piper_with_gripper_description.xacro", "end_effector_link": "gripper_base", "adapter_type": adapter_type, "address": address, @@ -68,7 +77,7 @@ def piper( "base_link": "base_link", "home_joints": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "base_pose": [0, y_offset, 0, 0, 0, 0, 1], - "package_paths": robot_asset_package_paths("piper"), + "package_paths": _PIPER_PACKAGE_PATHS, "xacro_args": {}, "auto_convert_meshes": True, "collision_exclusion_pairs": PIPER_GRIPPER_COLLISION_EXCLUSIONS, diff --git a/dimos/robot/catalog/ufactory.py b/dimos/robot/catalog/ufactory.py index 26787b9b95..68facb1978 100644 --- a/dimos/robot/catalog/ufactory.py +++ b/dimos/robot/catalog/ufactory.py @@ -18,14 +18,15 @@ from typing import Any -from dimos.robot.assets.manager import ( - RobotAssetPath, - robot_asset_package_paths, - robot_asset_xacro_args, -) +from dimos.robot.assets.source import RobotDescriptionSource from dimos.robot.config import GripperConfig, RobotConfig from dimos.utils.data import LfsPath +XARM_ROS2_REPO = "https://github.com/xArm-Developer/xarm_ros2" +_XARM_REPO = RobotDescriptionSource(url=XARM_ROS2_REPO, ref="humble") +_XARM_MODEL_PATH = _XARM_REPO / "xarm_description" / "urdf" / "xarm_device.urdf.xacro" +_XARM_PACKAGE_PATHS = {"xarm_description": _XARM_REPO / "xarm_description"} + # Pre-built URDFs for Pinocchio FK. The upstream xarm_ros2 source provides # Xacro-only model files, so these generated FK URDFs intentionally stay on LFS. XARM6_FK_MODEL = LfsPath("xarm_description/urdf/xarm6/xarm6.urdf") @@ -71,7 +72,9 @@ def xarm7( **overrides: Any, ) -> RobotConfig: """Create an xArm7 robot configuration.""" - xacro_args = robot_asset_xacro_args("xarm7") | { + xacro_args = { + "dof": "7", + "limited": "true", "attach_xyz": f"{x_offset} {y_offset} {z_offset}", "attach_rpy": f"0 {pitch} 0", } @@ -80,7 +83,7 @@ def xarm7( defaults: dict[str, Any] = { "name": name, - "model_path": RobotAssetPath("xarm7", "urdf"), + "model_path": _XARM_MODEL_PATH, "end_effector_link": "link_tcp" if add_gripper else "link7", "adapter_type": adapter_type, "address": address, @@ -88,7 +91,7 @@ def xarm7( "base_link": "link_base", "home_joints": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "base_pose": [x_offset, y_offset, z_offset, 0, 0, 0, 1], - "package_paths": robot_asset_package_paths("xarm7"), + "package_paths": _XARM_PACKAGE_PATHS, "xacro_args": xacro_args, "auto_convert_meshes": True, "collision_exclusion_pairs": XARM_GRIPPER_COLLISION_EXCLUSIONS if add_gripper else [], @@ -121,7 +124,9 @@ def xarm6( **overrides: Any, ) -> RobotConfig: """Create an xArm6 robot configuration.""" - xacro_args = robot_asset_xacro_args("xarm6") | { + xacro_args = { + "dof": "6", + "limited": "true", "attach_xyz": f"{x_offset} {y_offset} {z_offset}", "attach_rpy": f"0 {pitch} 0", } @@ -130,7 +135,7 @@ def xarm6( defaults: dict[str, Any] = { "name": name, - "model_path": RobotAssetPath("xarm6", "urdf"), + "model_path": _XARM_MODEL_PATH, "end_effector_link": "link_tcp" if add_gripper else "link6", "adapter_type": adapter_type, "address": address, @@ -138,7 +143,7 @@ def xarm6( "base_link": "link_base", "home_joints": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0], "base_pose": [x_offset, y_offset, z_offset, 0, 0, 0, 1], - "package_paths": robot_asset_package_paths("xarm6"), + "package_paths": _XARM_PACKAGE_PATHS, "xacro_args": xacro_args, "auto_convert_meshes": True, "collision_exclusion_pairs": XARM_GRIPPER_COLLISION_EXCLUSIONS if add_gripper else [], diff --git a/docs/capabilities/manipulation/a750.md b/docs/capabilities/manipulation/a750.md index 29e09ee2bd..dee66f3c83 100644 --- a/docs/capabilities/manipulation/a750.md +++ b/docs/capabilities/manipulation/a750.md @@ -59,10 +59,10 @@ The robot catalog entry is defined in [`dimos/robot/catalog/a750.py`](/dimos/rob | Base link | `base_link` | | End-effector link | `gripper_base` | | Home joints | `[0, 0, -90 deg, 0, 0, 0]` | -| Drake model | `RobotAssetPath("a750", "urdf")` → `urdf/a750_rev1.urdf` from `https://github.com/adob/a750_description` | +| Drake model | `RobotDescriptionSource(..., ref="master") / "urdf" / "a750_rev1.urdf"` from `https://github.com/adob/a750_description` | | FK/keyboard model | `a750_description/urdf/a750_rev1_no_gripper.urdf` | -The runtime model and `a750_description` package root are resolved through the Robot Asset Manager cache. The no-gripper model remains LFS-backed for Pinocchio FK because the upstream source publishes the full gripper URDF but not the generated no-gripper variant used by keyboard teleop. +The runtime model and `a750_description` package root are resolved through a Git-backed robot description source handle. The no-gripper model remains LFS-backed for Pinocchio FK because the upstream source publishes the full gripper URDF but not the generated no-gripper variant used by keyboard teleop. ## Gripper diff --git a/docs/capabilities/manipulation/adding_a_custom_arm.md b/docs/capabilities/manipulation/adding_a_custom_arm.md index 508cb637a5..2b32418755 100644 --- a/docs/capabilities/manipulation/adding_a_custom_arm.md +++ b/docs/capabilities/manipulation/adding_a_custom_arm.md @@ -497,21 +497,24 @@ If you want motion planning (collision-free trajectories via Drake), you need a ### 4a. Add your URDF -Prefer an upstream Robot Description Source and add a typed declaration in `dimos/robot/assets/declarations.py`. `RobotAssetPath` is a lazy `Path`-like adapter: importing the catalog does not clone or update the repo, but the first concrete path access resolves the source into `~/.cache/dimos/robot_assets`. +Prefer an upstream Robot Description Source and create a `RobotDescriptionSource` handle beside your robot adapter. Joining paths from the source is lazy: importing the catalog does not clone or update the repo, but the first concrete path access resolves the source into the robot asset cache. Use `LfsPath` only when the asset is intentionally vendored, locally modified, or has no suitable upstream source. ```python skip -from dimos.robot.assets.manager import RobotAssetPath, robot_asset_package_paths +from dimos.robot.assets.source import RobotDescriptionSource from dimos.manipulation.manipulation_module import manipulation_module from dimos.manipulation.planning.spec import RobotModelConfig from dimos.msgs.geometry_msgs.PoseStamped import PoseStamped from dimos.msgs.geometry_msgs.Quaternion import Quaternion from dimos.msgs.geometry_msgs.Vector3 import Vector3 -# Add a RobotAssetDeclaration for "yourarm" in dimos/robot/assets/declarations.py. -# Common artifact roles are "urdf", "mjcf", "srdf", and "mesh_dir". -_YOURARM_URDF_PATH = RobotAssetPath("yourarm", "urdf") +_YOURARM_REPO = RobotDescriptionSource( + url="https://github.com/example/yourarm_description", + ref="main", +) +_YOURARM_URDF_PATH = _YOURARM_REPO / "urdf" / "yourarm.urdf.xacro" +_YOURARM_PACKAGE_PATHS = {"yourarm_description": _YOURARM_REPO / "."} def _make_base_pose(x=0.0, y=0.0, z=0.0) -> PoseStamped: @@ -550,7 +553,7 @@ def _make_yourarm_config( joint_names=joint_names, end_effector_link="link6", # Last link in your URDF's kinematic chain base_link="base_link", # Root link of your URDF - package_paths=robot_asset_package_paths("yourarm"), + package_paths=_YOURARM_PACKAGE_PATHS, xacro_args={}, # Xacro arguments if using .xacro files collision_exclusion_pairs=[], # Pairs of links that can touch (e.g., gripper fingers) auto_convert_meshes=True, # Convert DAE/STL meshes for Drake diff --git a/docs/capabilities/manipulation/readme.md b/docs/capabilities/manipulation/readme.md index e31ae048bd..18c534e80b 100644 --- a/docs/capabilities/manipulation/readme.md +++ b/docs/capabilities/manipulation/readme.md @@ -192,11 +192,9 @@ visualization backend. ## Robot model assets -XArm6, XArm7, Piper, and A-750 runtime model paths are resolved through the Robot Asset Manager (`dimos.robot.assets`). Catalogs use `RobotAssetPath` so imports stay lightweight: no network or Git work happens until a concrete path is accessed. +XArm6, XArm7, Piper, and A-750 runtime model paths are resolved through Git-backed robot description source handles (`dimos.robot.assets`). Catalogs join relative paths from a `RobotDescriptionSource`, so imports stay lightweight: no network or Git work happens until a concrete path is accessed. -Declared upstream sources are cached under `~/.cache/dimos/robot_assets`. The cache is fresh-when-safe: a missing checkout is cloned, a clean checkout is updated, update failures warn and continue with the cached copy, and dirty local checkouts are preserved with a warning. - -Common artifact roles are flat strings: `urdf`, `mjcf`, `srdf`, and `mesh_dir`. Extra roles such as `urdf_ik` may be declared for robot-specific planning/FK needs. +Upstream sources are cached under the platform user cache directory in `dimos/robot_assets`. The cache is fresh-when-safe: a missing checkout is cloned, a clean checkout is updated, update failures warn and continue with the cached copy, and dirty local checkouts are preserved with a warning. Some FK-only Pinocchio assets remain on LFS when the upstream source does not publish a static no-gripper URDF/MJCF equivalent. OpenArm also remains on the existing path because DimOS carries local model modifications. diff --git a/docs/coding-agents/code-quality-rules.md b/docs/coding-agents/code-quality-rules.md index 67bee4ce95..d59d543ccd 100644 --- a/docs/coding-agents/code-quality-rules.md +++ b/docs/coding-agents/code-quality-rules.md @@ -16,7 +16,7 @@ Rules dimos code is expected to follow. They address recurring issues found in c * Specify only what differs from defaults. Don't restate defaults like `tick_rate=100.0`, `publish_joint_state=True`, or default topics (`/cmd_vel`, `/odom`). * `.transports({...})` applies to all matching modules, so define a remap once, not twice across sub-blueprints. * No lambdas -- they can't be pickled to worker processes. Use named functions. -* Do no work at import time: no subprocesses, viewers, model parsing, or network. In particular don't call `get_data(...)` (it blocks import until the download finishes) -- use lazy path adapters such as `RobotAssetPath` for upstream robot descriptions or `LfsPath` for intentionally vendored assets, or build the config in `start`/`build`. Any process you start must be managed (shut down when not needed). +* Do no work at import time: no subprocesses, viewers, model parsing, or network. In particular don't call `get_data(...)` (it blocks import until the download finishes) -- use lazy path adapters such as `RobotDescriptionSource` paths for upstream robot descriptions or `LfsPath` for intentionally vendored assets, or build the config in `start`/`build`. Any process you start must be managed (shut down when not needed). * Blueprint files define blueprints, not modules/classes. * Helper blueprints not meant to run alone must start with `_` (the `all_blueprints.py` generator skips them); demo/non-shared ones get a `demo_` prefix (hidden from `dimos list`). diff --git a/pyproject.toml b/pyproject.toml index b0570ec70a..59c1ce16c1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -110,6 +110,7 @@ dependencies = [ "plum-dispatch==2.5.7", "GitPython>=3.1.45", "filelock>=3.20.0", + "platformdirs>=4.0.0", # Logging "structlog>=25.5.0,<26", "colorlog==6.9.0", diff --git a/uv.lock b/uv.lock index 0def8d02fd..fc634abe21 100644 --- a/uv.lock +++ b/uv.lock @@ -1903,6 +1903,7 @@ dependencies = [ { name = "open3d-unofficial-arm", marker = "platform_machine == 'aarch64' and sys_platform == 'linux'" }, { name = "opencv-python" }, { name = "pin" }, + { name = "platformdirs" }, { name = "plotext" }, { name = "plum-dispatch" }, { name = "protobuf" }, @@ -2447,6 +2448,7 @@ requires-dist = [ { name = "pin", specifier = ">=3.3.0" }, { name = "pin-pink", marker = "extra == 'manipulation'", specifier = ">=4.2.0" }, { name = "piper-sdk", marker = "extra == 'manipulation'" }, + { name = "platformdirs", specifier = ">=4.0.0" }, { name = "playground", marker = "extra == 'sim'", specifier = ">=0.0.5" }, { name = "plotext", specifier = "==5.3.2" }, { name = "plotly", marker = "extra == 'manipulation'", specifier = ">=5.9.0" }, From 9bd1354109285c60b9babc3f998966269675cb34 Mon Sep 17 00:00:00 2001 From: cc Date: Fri, 19 Jun 2026 19:44:59 -0700 Subject: [PATCH 13/15] fix: satisfy robot asset mypy --- dimos/robot/assets/source.py | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/dimos/robot/assets/source.py b/dimos/robot/assets/source.py index 3a93298da6..c5e18a3bdd 100644 --- a/dimos/robot/assets/source.py +++ b/dimos/robot/assets/source.py @@ -137,13 +137,15 @@ def __truediv__(self, other: object) -> RobotDescriptionPath: def __eq__(self, other: object) -> bool: if not isinstance(other, RobotDescriptionPath): return self._resolve() == other - return object.__getattribute__( - self, "_robot_description_source" - ) == object.__getattribute__( + source: RobotDescriptionSource = object.__getattribute__(self, "_robot_description_source") + other_source: RobotDescriptionSource = object.__getattribute__( other, "_robot_description_source" - ) and object.__getattribute__( - self, "_robot_description_relative_path" - ) == object.__getattribute__(other, "_robot_description_relative_path") + ) + relative_path: Path = object.__getattribute__(self, "_robot_description_relative_path") + other_relative_path: Path = object.__getattribute__( + other, "_robot_description_relative_path" + ) + return source == other_source and relative_path == other_relative_path def __hash__(self) -> int: return hash( From e24ccbdc57ca7d72eb144eb7603e10b561ca3e74 Mon Sep 17 00:00:00 2001 From: cc Date: Fri, 19 Jun 2026 20:23:35 -0700 Subject: [PATCH 14/15] chore: remove stuff --- CONTEXT.md | 37 ---------- dimos/utils/test_git_asset_cache.py | 111 ---------------------------- 2 files changed, 148 deletions(-) delete mode 100644 CONTEXT.md delete mode 100644 dimos/utils/test_git_asset_cache.py diff --git a/CONTEXT.md b/CONTEXT.md deleted file mode 100644 index cdab58e9bb..0000000000 --- a/CONTEXT.md +++ /dev/null @@ -1,37 +0,0 @@ -# DimOS Robotics - -DimOS composes robot software from reusable modules and robot-specific descriptions. This context defines the language used when discussing robot model assets and how DimOS consumes them. - -## Language - -**Robot Description Source**: -An upstream repository that contains a robot's URDF, Xacro, MJCF, meshes, and related package files. -_Avoid_: URDF repo, asset repo, model repo - -**Robot Asset Manager**: -A DimOS-facing service that resolves robot description sources into local filesystem paths for use by robot model consumers. -_Avoid_: LFS replacement, description downloader, asset loader - -**Robot Asset Cache**: -The standard user cache directory where DimOS stores fetched robot description source checkouts for reuse across runs. -_Avoid_: Data directory, vendored assets, repository assets - -**Robot Asset Supply Path**: -The source and resolution route by which DimOS obtains robot model assets for both built-in and user-supplied robots. -_Avoid_: Built-in asset path, custom asset path, deployment path - -**Robot Description Source Handle**: -A path-like object that names a robot description source and resolves to the local checkout root for that source. Callers form concrete model paths by joining relative paths from this root. -_Avoid_: Registry entry, artifact lookup, asset manifest - -**ROS Package Root**: -The local directory corresponding to a ROS-style package name, used to resolve `package://...` URIs and `$(find package_name)` expressions in robot model files. -_Avoid_: Package path, asset package, Python package - -**DimOS Robot Model Config**: -A DimOS configuration object that names the model paths, package paths, joints, links, and robot-specific metadata needed by planning, control, simulation, or visualization. -_Avoid_: Robot description, URDF config - -**Registered Description Module**: -An importable description entry provided by a third-party robot description registry. -_Avoid_: Robot description source, GitHub repo diff --git a/dimos/utils/test_git_asset_cache.py b/dimos/utils/test_git_asset_cache.py deleted file mode 100644 index b8a01e1c90..0000000000 --- a/dimos/utils/test_git_asset_cache.py +++ /dev/null @@ -1,111 +0,0 @@ -# Copyright 2025-2026 Dimensional Inc. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from pathlib import Path -import shutil -import subprocess - -import pytest - -from dimos.robot.assets.git_cache import GitAssetCache, GitAssetCacheWarning - - -def _git(cwd: Path, *args: str) -> str: - return subprocess.run( - ["git", *args], - cwd=cwd, - check=True, - capture_output=True, - text=True, - ).stdout.strip() - - -def _commit(repo: Path, relative_path: str, contents: str, message: str) -> str: - path = repo / relative_path - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(contents) - _git(repo, "add", relative_path) - _git(repo, "commit", "-m", message) - return _git(repo, "rev-parse", "HEAD") - - -@pytest.fixture() -def local_repo(tmp_path: Path) -> Path: - repo = tmp_path / "upstream" - repo.mkdir() - _git(repo, "init", "-b", "main") - _git(repo, "config", "user.email", "test@example.com") - _git(repo, "config", "user.name", "Test User") - _commit(repo, "asset.txt", "v1", "initial") - return repo - - -def test_clone_on_miss_resolves_local_git_repo_at_branch(local_repo: Path, tmp_path: Path) -> None: - cache = GitAssetCache(tmp_path / "cache") - - checkout = cache.resolve(str(local_repo), "main") - - assert checkout.updated is True - assert checkout.path.exists() - assert (checkout.path / "asset.txt").read_text() == "v1" - assert _git(checkout.path, "rev-parse", "--abbrev-ref", "HEAD") == "main" - - -def test_clean_cached_repo_updates_when_upstream_branch_changes( - local_repo: Path, tmp_path: Path -) -> None: - cache = GitAssetCache(tmp_path / "cache") - first = cache.resolve(str(local_repo), "main") - first_commit = _git(first.path, "rev-parse", "HEAD") - second_commit = _commit(local_repo, "asset.txt", "v2", "update") - - second = cache.resolve(str(local_repo), "main") - - assert second.path == first.path - assert second.updated is True - assert _git(second.path, "rev-parse", "HEAD") == second_commit - assert _git(second.path, "rev-parse", "HEAD") != first_commit - assert (second.path / "asset.txt").read_text() == "v2" - - -def test_dirty_cached_repo_skips_update_and_preserves_local_edits( - local_repo: Path, tmp_path: Path -) -> None: - cache = GitAssetCache(tmp_path / "cache") - checkout = cache.resolve(str(local_repo), "main") - (checkout.path / "asset.txt").write_text("local edit") - _commit(local_repo, "asset.txt", "upstream edit", "upstream update") - - with pytest.warns(GitAssetCacheWarning, match="local changes"): - dirty_checkout = cache.resolve(str(local_repo), "main") - - assert dirty_checkout.skipped_dirty_update is True - assert (dirty_checkout.path / "asset.txt").read_text() == "local edit" - - -def test_clean_cached_repo_returns_cached_checkout_when_fetch_fails( - local_repo: Path, tmp_path: Path -) -> None: - cache = GitAssetCache(tmp_path / "cache") - checkout = cache.resolve(str(local_repo), "main") - cached_commit = _git(checkout.path, "rev-parse", "HEAD") - shutil.rmtree(local_repo) - - with pytest.warns(GitAssetCacheWarning, match="using cached checkout"): - fallback = cache.resolve(str(local_repo), "main") - - assert fallback.used_cached_fallback is True - assert fallback.path == checkout.path - assert _git(fallback.path, "rev-parse", "HEAD") == cached_commit - assert (fallback.path / "asset.txt").read_text() == "v1" From 2995d42ff01d817d3e7c74b1098023b9dbf1c98e Mon Sep 17 00:00:00 2001 From: cc <55869557+TomCC7@users.noreply.github.com> Date: Fri, 19 Jun 2026 20:26:54 -0700 Subject: [PATCH 15/15] Update readme.md --- docs/capabilities/manipulation/readme.md | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/capabilities/manipulation/readme.md b/docs/capabilities/manipulation/readme.md index 18c534e80b..d991cb3f21 100644 --- a/docs/capabilities/manipulation/readme.md +++ b/docs/capabilities/manipulation/readme.md @@ -190,14 +190,6 @@ visualization backend. | XArm6 | 6 | Y | Y | — | | XArm7 | 7 | Y | Y | Y | -## Robot model assets - -XArm6, XArm7, Piper, and A-750 runtime model paths are resolved through Git-backed robot description source handles (`dimos.robot.assets`). Catalogs join relative paths from a `RobotDescriptionSource`, so imports stay lightweight: no network or Git work happens until a concrete path is accessed. - -Upstream sources are cached under the platform user cache directory in `dimos/robot_assets`. The cache is fresh-when-safe: a missing checkout is cloned, a clean checkout is updated, update failures warn and continue with the cached copy, and dirty local checkouts are preserved with a warning. - -Some FK-only Pinocchio assets remain on LFS when the upstream source does not publish a static no-gripper URDF/MJCF equivalent. OpenArm also remains on the existing path because DimOS carries local model modifications. - ## Adding a Custom Arm [guide is here](/docs/capabilities/manipulation/adding_a_custom_arm.md)