Jian Zhou · Sihao Lin · Jin Li · Shuai Fu · Gengze Zhou · Qi Wu
Australian Institute for Machine Learning, University of Adelaide
Recorded live in the editor — the MapGPT executor loads, then a real R2R episode runs end-to-end.
A visual agent-design platform for embodied AI research. One typed graph, two roles: a harness that runs embodied agents, and a scaffold that coding agents edit and verify.
AgentCanvas lets researchers prototype embodied agents — for VLN, EQA, VLA, and adjacent tasks — by drawing node graphs that execute in real time against simulators (Habitat-Sim, MatterSim, SAPIEN/ManiSkill2, MuJoCo/robosuite) or, in principle, real-world setups. One JSON = one agent = one graph: agent behaviour is a dataflow graph, not imperative code; the graph is the source of truth, saved as a single JSON file and loaded as a complete agent.
Built for: researchers who want to compose, compare, and share embodied agent architectures without rewriting the execution stack each time. The platform covers VLN (Vision-and-Language Navigation), EQA (Embodied Question Answering), VLA (Vision-Language-Action) policy benchmarks, and adapts to other embodied / agentic settings via the nodeset model.
Status: research preview, pre-1.0 — 40+ nodesets across four swappable palettes (env · method · model · policy); the public API is not yet frozen (Versioning Policy).
Contributing: nodesets, graphs, and core PRs all welcome — every contribution is credited on the Credits board. See CONTRIBUTING.md.
-
[2026/07] 🎥 VGGT-SLAM 2 (RSS 2026) — dense feed-forward RGB SLAM by Dominic Maggio & Luca Carlone (MIT-SPARK, BSD-2-Clause), ported to AgentCanvas as a nodeset. On TUM RGB-D, RGB-only — no depth, no intrinsics: keyframes batch into submaps, one VGGT feed-forward pass reconstructs each, and a GTSAM SL(4) projective pose-graph fuses them. The clip orbits the finished ~9.3M-point dense map, then shows the live in-canvas build — the camera trajectory arriving in submap bursts and the dense map growing chunk-by-chunk. Full walkthrough on the VGGT-SLAM 2 nodeset docs.
-
[2026/07] 🚀 Graph SDK — build & run agents in Python — the same canvas graphs, now an importable library:
from agentcanvas import Graph, add/connect nodes, run and batch-eval in-process, or compile a graph back into a standalone builder script. SameGraphDefinition, fully reversible with canvas + JSON. See the Graph SDK docs. -
[2026/07] 🎥 pySLAM — classic feature-based visual SLAM by Luigi Freda (GPL-3.0), wrapped in AgentCanvas as a nodeset behind a container bridge. On TUM RGB-D: a streaming-replay env feeds a benchmark sequence frame-by-frame into a live SLAM session — the estimated camera trajectory fitted onto ground truth top-down and a sparse 3-D map densifying in real time, no simulator or policy, CPU-only. Full walkthrough on the pySLAM nodeset docs.
-
[2026/07] 🔥 Broader foundation-model support — 29 foundation models are now wired as thin server-mode shells (transformers-native + other sources), available to both hand-built graphs and the AAS optimizer: recent VLMs (Qwen3-VL, InternVL3, Gemma 3, SmolVLM2), open-vocabulary perception (SigLIP2, OWLv2, Grounding DINO), and geometry / depth backbones. See the foundation-model coverage and per-model Credits.
-
[2026/07] 🔥 Edit node source from the canvas — the new Source tab shows the selected node's scoped slice of its nodeset source (globals, referenced functions, the class itself) and splices edits back with syntax-checked hot-reload. PR: #5.
-
[2026/07] 🎉 First public release — AgentCanvas is open-sourced as a research preview (pre-1.0). Docs: jianzhou0420.github.io/AgentCanvas.
- Why AgentCanvas? — a searchable substrate for embodied agents, and the pain points it solves
- Features — the one JSON = one agent (§2.2) / one Python class = one node (§2.6) principles, plus canvas editor, graph executor, isolated runtime envs, nested graphs, state containers, hooks
- Sim-to-Real Path — same agent graph on simulator today, real robot tomorrow — via env-as-nodeset + server mode + ROS
- Getting Started — prerequisites, run the web dashboard, run an evaluation, run architecture search, serve the docs
- Contributing — where help is most wanted · credits
- Citation — cite the AgentCanvas paper
- License — Apache 2.0
Embodied agents — spanning VLN, EQA, and VLA — are increasingly built by composing foundation models with perception, mapping, memory, planning, and action. Unlike end-to-end policies, whose structure is absorbed into weights, this architecture is explicit and editable. That raises the question AgentCanvas is built around — can agent design be searched rather than hand-built? — alongside two stacks of pain it has to clear on the way.
Agent architecture is hand-built — and could be searched
Each agent fixes a choice at every join — sensor abstractions, map representations, memory state, prompt structure, planner topology, model placement, action interfaces — by hand, usually for a single benchmark. As foundation models and embodied tools multiply, the space grows faster than manual iteration can cover, so the natural move is to search it rather than hand-tune it.
Agent Architecture Search (AAS) already does this for text-domain agents, but embodied transfer is not free: stateful simulators, noisy multi-episode scoring, long perception/action traces, and no off-the-shelf palette of embodied primitives. AgentCanvas is our attempt to supply the missing substrate — a scaffold a coding agent can read, edit, run, and verify — so that searching agent design becomes possible for embodied agents too.
Embodied-specific pain points
-
Modern embodied stack is thick — a working embodied agent needs LLM reasoning + tool use + simulator coupling + spatial tools, all wired together. Building this from scratch per project is prohibitively costly, and most of the effort goes into the execution layer rather than the idea being tested.
-
Engineering nightmare — an embodied agent is not one model but a whole system — a stateful simulator plus a stack of heavy models and tools. Just running it, let alone at the scale benchmarking needs, is a hard engineering job in itself:
- Python env hell — no single Python env satisfies every part; each simulator, VLM, detector, and policy pins its own clashing CUDA / torch / Python, so finding one runtime they all share is often impossible — you end up maintaining several incompatible environments just to load the agent.
- Batching — each worker's simulator is a separate stateful process advancing at its own pace; you can batch the model but not the sims, so every step becomes an async gather-observations → batch-infer → scatter-actions dance.
- Other infra — multimodal trajectories that must be logged and replayable, checkpoint/resume of multi-hour GPU runs that will crash, and debugging across process boundaries.
Over a single paper's research cycle, the researcher pays far too much of this engineering cost instead of focusing on the algorithm itself.
-
Hidden ground-truth dependencies — many methods quietly rely on simulator-provided ground truth (object poses, semantic labels, navigability) rather than real perception. Sometimes that's a legitimate way to control the experiment — but, oversight or not, it often goes unmentioned in the paper.
Common AI research pain points (amplified here)
- Non-reproducible implementations — every paper builds its agent from scratch with a different codebase; comparing methods fairly or reproducing results is painful — and many of them are
Code coming SOON(Someday, Or Obviously Never). - Paper ≠ code — papers show clean flow diagrams, but the actual code diverges in undocumented ways. Reproducing a paper means reverse-engineering its implementation.
- Tightly coupled code — domain logic (prompts, tools, policies) is tangled with infrastructure. Swapping one component means rewriting the pipeline.
Full reference in the docs — most features below have an implementation page (mechanism · key files · current status): The Nine Capabilities →
The nine capabilities — canvas editor · graph engine · isolated runtimes · nested graphs · state containers · Python-defined nodes · hooks · batch eval · observability
A ComfyUI-style flat workspace where all node types coexist — environments, LLMs, reasoning chains, control gates, and output viewers. Drag nodes from the sidebar, wire them together, hit Play.
One JSON = one agent. An agent's entire behavior — nodes, wiring, configs, state containers, hooks — is a single JSON file: load it, run it, share it, diff it. No hidden pipeline code; what you see on the canvas is what executes.
The engine then runs that graph: nodes fire when their inputs arrive, not in a fixed order. The same engine handles every graph shape AgentCanvas v1 supports — see major-versions.html for the full list of agent forms covered by v1's bounded-static-topology paradigm:
- DAG workflows — single forward pass for acyclic pipelines
- Cyclic agent loops — observe-think-act-repeat via a two-pivot model: a two-sided
IterIn(run-start init inputs on its left, per-iteration loop-carry on its right) plusIterOut, keeping the graph visually acyclic while enabling runtime cycles (ADR-dataflow-008, which folded the earlier three-pivotinitialize/IterIn/IterOut of ADR-dataflow-006 down to two) - Multi-scope iteration — N coexisting
(IterIn, IterOut)pairs in one flat graph (ADR-dataflow-007 / ADR-executor-003) - ReAct loops — either hidden inside an
LLMCallNodesubclass or expressed explicitly as router + N predeclared tool branches - Bounded multi-agent — fixed-N or
K_max-bounded fan-out (e.g., DiscussNav-style debate, AutoGen-style fixed roles) - Plan-and-Execute — over a bounded tool pool, dispatched by router
The engine is also extensible without touching graph nodes: shell hooks fire before/after each node execution and at graph lifecycle boundaries — log outputs, validate inputs, block nodes, or modify data — and travel with saved graphs.
Research tools often need conflicting Python environments (Habitat needs Python 3.8, SLAM needs ROS). Any BaseNodeSet can run in server mode — the framework auto-generates an HTTP server from the nodeset's port definitions, running in its own interpreter. Zero extra code:
# Same nodeset code, two deployment modes:
POST /api/components/nodesets/env_habitat/load # in-process
POST /api/components/nodesets/env_habitat/load?mode=server # separate process
Save any canvas graph as a graph node and drag it onto another canvas as a reusable block. This enables hierarchical agent architectures — a high-level planner containing sub-agent graph nodes. Snapshot semantics: each instance is a deep copy.
Shared persistent state across agent loop iterations via a dual-wire architecture:
- Data edges carry dataflow between nodes (IMAGE, TEXT, ACTION, POSE, …)
- Access grants let nodes read/write StateContainers — visible canvas elements with named entries, configurable reducers (Accumulator, LastWrite, Counter), and a Lifetime axis (
forever/step/episode/run/custom) that auto-clears memory on the right signal boundary (ADR-dataflow-002, ADR-dataflow-004)
One Python class = one node. Every canvas node — tools, environments, skills, policies — is a single Python class: declare ports, implement forward(), drop the file in workspace/, and the platform auto-discovers it. No framework changes, no TypeScript, no registration boilerplate.
from app.components import BaseCanvasNode, PortDef
class MeasureDistanceNode(BaseCanvasNode):
node_type = "basic_agent__measure_distance"
display_name = "Measure Distance"
description = "Euclidean distance between two 3D positions"
category = "tool"
icon = "Ruler"
input_ports = [
PortDef("pos_a", "TEXT", "Position A as [x, y, z]"),
PortDef("pos_b", "TEXT", "Position B as [x, y, z]"),
]
output_ports = [
PortDef("distance", "TEXT", "Euclidean distance (meters)"),
]
async def forward(self, inputs, ctx):
a, b = parse_vec3(inputs["pos_a"]), parse_vec3(inputs["pos_b"])
dist = math.sqrt(sum((ai - bi) ** 2 for ai, bi in zip(a, b)))
return {"distance": f"{dist:.2f}"}The node then appears in the canvas sidebar and wires to any other node with matching port types. Its appearance is Python-driven too: GenericBlockRenderer renders any node automatically from NodeUIConfig — colors, layout, inline config controls (sliders, dropdowns, text fields), and display widgets — so no custom React component is needed.
The same graph that runs on the canvas can be submitted as an eval job that scores it over hundreds of episodes. A backend-owned JobScheduler gates admission against a VRAM budget shared across all sessions (ADR-eval-003); each admitted run is its own subprocess whose lifetime is tied to the backend (PR_SET_PDEATHSIG) — no orphaned GPU processes, and every finished episode persists on disk. Per-episode logs land in a self-contained layout (ADR-eval-004) so a teammate can replay any single episode without re-running.
Every step streams observations, reasoning, actions, and metrics via WebSocket, routed by execution_id so concurrent runs don't cross streams. Errors from any source — node exceptions, server-mode subprocess crashes, and HTTP failures — flow through a unified ErrorBus and surface as Report-tab entries + toasts (ADR-observability-004). (React render errors are caught by a client-side error boundary.)
AgentCanvas is designed for portability: a single agent graph can execute against a simulator today and migrate to a real robot in the future without graph-level changes. This property follows from two architectural decisions — environments are themselves nodesets (ADR-components-002), and any nodeset can execute in an isolated runtime via server mode (ADR-server-001).
The full path — today's simulators · a ROS nodeset with the same interface · bidirectional integration · ground-truth visibility
The shipped environments — Habitat (VLN-CE), MatterSim / MP3D, HM3D ObjectNav, HM-EQA, OpenEQA, SIMPLER (real-to-sim VLA), and LIBERO (manipulation) — are each implemented as a BaseNodeSet that exposes observation and action ports. The agent graph connects to these ports and never imports the simulator directly, which keeps the graph independent of any specific environment implementation.
Real-robot deployment is achieved by replacing the simulator nodeset with a ROS nodeset that exposes the same observation / act interface. Internally, this nodeset composes existing ROS components — cv_bridge, Nav2, MoveIt, and hardware driver packages — into a unified facade. Server mode launches the nodeset inside its own ROS Python environment and bridges it to the canvas over HTTP. The agent graph itself is unchanged.
This division of labor is favorable because the substantive engineering — perception, control, motion planning, and hardware interfacing — already exists as mature ROS packages. The ROS-side adapter is therefore a composition task rather than greenfield development, and the AgentCanvas-side env nodeset reduces to a thin HTTP client.
The boundary between AgentCanvas and ROS is symmetric; either side may own the control loop:
- ROS as a subsystem of AgentCanvas (native pattern; server mode is designed for this case) — the ROS nodeset runs in server mode, AgentCanvas drives the agent loop, and ROS provides sensing and actuation.
- AgentCanvas as a subsystem of ROS (also supported; no framework modifications required) — when the broader project is ROS-led, the ROS-side control loop invokes AgentCanvas's
/runendpoint at each step (treating the graph as a policy) and publishes the returned action. This requires only a thin ROS bridge node on the ROS side.
The same nodeset abstraction directly addresses two pain points raised in §1. A node that queries simulator ground truth (e.g., env_habitat__get_object_pose) and a node that performs real perception (e.g., a SAM-based detector) appear as visibly distinct blocks on the canvas. Whether an agent depends on ground truth or on perception is therefore a property of the graph topology, not a hidden implementation detail. Substituting one for the other is a local edge change, not a code refactor.
All currently shipped environment nodesets are simulator-based. A real-robot ROS nodeset remains a call-for-contribution slot — the architectural pathway is established and intentional, and the necessary ROS-side components are already available in the ecosystem.
There are two ways to use AgentCanvas, both over the same typed-graph substrate:
- Build & run a graph by hand — compose nodes on the canvas, run an agent live against a simulator, and evaluate it at scale (the rest of this section).
- Agent Architecture Search (AAS) — hand a seed graph to a coding agent and let it search architectures for you (jump).
- Python 3.10+ with Conda (the default
agentcanvasenv — ADR-platform-004) - Node.js 18+
- (Optional, for Habitat-Sim) a separate Python 3.8 env —
habitat-sim 0.1.7only runs here; AgentCanvas talks to it via server mode, see INSTALL.md
# Activate environment
conda activate agentcanvas
# Start backend (FastAPI :8000) + frontend (Vite :5173)
cd agentcanvas && bash run_dev.shOpen http://localhost:5173 to access the canvas editor.
The same eval pipeline is exposed through four interfaces — pick by what you're holding:
| # | Interface | Audience | Best for |
|---|---|---|---|
| 1 | Frontend Eval page | Human | Click-driven, watch live progress in the UI |
| 2 | /experiment:run slash command |
Coding agent (Claude Code) | Profile-gated GPU admission, auto-allocated port, no :8000 clobber |
| 3 | MCP server | Coding agent | Conversational, ad-hoc eval — no slash-command overhead |
| 4 | HTTP API | Scripts / CI | Direct REST, no MCP required |
Open a saved graph on the Eval page, pick a split + episode range, hit Start. Progress streams live over WebSocket; results land as per-episode JSONL under outputs/eval_runs/{run_id}/episodes/ep{NNNN}/ (ADR-eval-004) and are browsable in the Run Detail panel. Multi-worker env fan-out and batched inference are configurable from the form (ADR-eval-002).
When using Claude Code, /experiment:run <profile> -- <cmd> wraps any eval invocation in the backend's JobScheduler admission gate (ADR-eval-003): the wrapper claims VRAM under the declared profile in .claude/commands/experiment/profiles.yaml, spawns the backend on an allocated port (BACKEND_URL=http://127.0.0.1:<port> is exported to the wrapped command), and releases the slot on exit. Companion commands: /experiment:status for run snapshots, /experiment:teardown for graceful cancellation.
→ .claude/commands/experiment/README.md
For full architecture-search design loops (many iterations of propose → evaluate → keep-the-best over a seed graph), see Run Agent Architecture Search below.
Register agentcanvas-backend with any MCP-aware client (Claude Code, Cursor, …) and call typed tools (graph_list, eval_start, eval_status, eval_export, eval_stop) conversationally. No iter-tree bookkeeping — just raw eval against a borrow-or-spawned backend.
→ agentcanvas/mcp_server/README.md
Direct REST for scripts, CI, or non-MCP environments:
curl -X POST http://localhost:8000/api/eval/v2/start \
-H 'content-type: application/json' \
-d '{"graph_name": "navgpt_ce", "split": "val_unseen", "worker_count": 4}'
# poll GET /api/eval/v2/status
# fetch GET /api/eval/v2/export/{run_id}→ Driving the Backend from a Coding Agent — deep dive on all programmatic modes side-by-side
Beyond evaluating a fixed graph, AgentCanvas is the substrate for Agent Architecture Search — a development-time loop where an LLM coding-agent Optimizer repeatedly proposes graph edits to a seed Executor, evaluates each candidate in the simulator, and keeps the improvements (§1 — why a searchable substrate). Because an agent is a typed graph, each candidate is a type-checked patch run before any expensive rollout, and per-node episode logs let the Optimizer attribute score changes to specific modules.
The coding-agent optimizer searching over an embodied executor's graph — propose an edit, run it, keep the gains.
Search is method-seeded: iter_0 is a published embodied method and the loop searches graph-level edits around it. Three search variants ship as Claude Code skills under .claude/commands/architect/, sharing one coding-agent harness (proposer → implementer → evaluator) and differing only in proposer logic + persistent memory:
| Variant skill | Paper name | Search policy |
|---|---|---|
myloop |
KDLoop | Four-phase THINK → CRITIC → EXPERIMENT → DISTILL cycle, typed memory + REFLECT meta-phase |
adas-subagent |
ADAS (port) | Reflexion-style proposals over a flat append-only archive |
aflow |
AFlow (port) | Score-softmax parent selection + anti-replay memory |
# In a Claude Code session on this repo — run KDLoop over the MapGPT executor
/architect:myloop:loop mapgpt_mp3d --goal "raise val_unseen SR"
# The ADAS / AFlow ports take the same <graph> [<version>] form
/architect:adas-subagent:loop smartway_ce
/architect:aflow:loop explore_eqa_hmeqa
Seed graphs currently wired for search: mapgpt_mp3d, smartway_ce (VLN), explore_eqa_hmeqa (EQA), voxposer_libero_monolithic (VLA). Each iteration writes its proposal, patch, eval scores, and logs under outputs/design_runs/{variant}/{graph}/vN/iter_M/.
# Serve the doc-site locally on :8092 (live-reload via SSE)
bash docs/run_dev.shTwo kinds of contribution, both welcome — see CONTRIBUTING.md:
- Content — nodesets & graphs. Write a nodeset that wraps a tool / simulator / model (e.g. real-time 3D Gaussian Splatting, a voxel-based SLAM system) or encodes a method (e.g. NavGPT, MapGPT), or compose a graph that wires existing nodesets into a complete agent. Open a PR into
workspace/; review is light. - Core — UI, backend, framework. Bug fixes, new features, even refactors are welcome. The one ask: if a change is big enough to cost real time, open a Discussion first so we can align before you build.
Every nodeset and graph is credited to its author/maintainer on the board below — with a citation link if it has an associated paper — so contributing here doesn't cost you authorship. The AgentCanvas framework and the first-release methods, graphs, and environment integrations are by AC-Team. The foundation models and policies below are third-party — AgentCanvas ships only a thin server-mode wrapper so each plugs into the graph (for human users and the AAS optimizer alike); credit for every model belongs to its original authors — the foundation models are pulled into a separate table below, split by source (transformers-native vs. torch.hub / torchvision / vendored upstream repo), with full per-model attribution on the Credits page. The board is names-only by design: the canonical inventory with per-graph verification detail lives on the doc-site Credits page and the VLN / EQA / VLA support-status pages.
✅ verified — reproduces its paper / reference implementation · 🚧 runs end-to-end, verification in progress
| Environments | Methods | Models & Policies |
|---|---|---|
|
|
|
Foundation models — third-party models wrapped behind one thin nodeset shell (lazy load · single-flight GPU · base64-npy dataflow envelope) so each is a uniform building block for both human users and the AAS optimizer. We do not author these — we ship only the shell; credit belongs to the original authors (full per-model attribution + papers on the Credits page). Split by source:
transformers-native (thin wrapper over AutoModel / pipeline) |
Other sources (torch.hub / torchvision / vendored upstream repo) |
|---|---|
|
|
Call for contribution — reserved slots, credited to whoever lands them (how to contribute; IDs are roadmap slots on the Credits page):
| Benchmarks | Methods | Features & Infra |
|---|---|---|
|
|
|
If you use AgentCanvas in your research, please cite:
@misc{jian2026AgentCanvas,
title = {Automating the Design of Embodied Agent Architectures},
author = {Jian Zhou and Sihao Lin and Jin Li and Shuai Fu and Gengze Zhou and Qi Wu},
year = {2026},
eprint = {2606.30111},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2606.30111}
}Apache License 2.0 — see LICENSE.