From d52c478dc7a95361ec7e22cae222cb4ff82492b7 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 06:55:47 +0200 Subject: [PATCH 01/27] doc: add a one-bot guide and route the docs nav to it AI-assisted: Claude Opus 4.8 --- doc/guides/index.md | 9 +++++--- doc/guides/one-bot.md | 52 +++++++++++++++++++++++++++++++++++++++++++ doc/index.md | 4 +--- 3 files changed, 59 insertions(+), 6 deletions(-) create mode 100644 doc/guides/one-bot.md diff --git a/doc/guides/index.md b/doc/guides/index.md index bd10c944..cb5c1e9a 100644 --- a/doc/guides/index.md +++ b/doc/guides/index.md @@ -4,11 +4,14 @@ Task-oriented walkthroughs that span several commands. ```{toctree} :hidden: -lh2-calibration +one-bot controller +lh2-calibration ``` -- [Lighthouse 2 localization](lh2-calibration.md) - give your bots real-world - `(x, y)` positions. +- [Drive a single DotBot](one-bot.md) - build, flash, and control one bot end to + end. - [Run the controller + web UI](controller.md) - drive and visualize a swarm from the browser. +- [Lighthouse 2 localization](lh2-calibration.md) - give your bots real-world + `(x, y)` positions. diff --git a/doc/guides/one-bot.md b/doc/guides/one-bot.md new file mode 100644 index 00000000..6fe78cf3 --- /dev/null +++ b/doc/guides/one-bot.md @@ -0,0 +1,52 @@ +# Drive a single DotBot + +Build and flash one DotBot and a gateway, cable them to your computer, and drive +the bot from the web UI. This is the smallest real-hardware setup. For the +no-hardware path, use the [simulator](controller.md) instead. + +Building firmware needs SEGGER Embedded Studio and `nrfjprog` (see the README +prerequisites). To skip building, fetch a pre-built release with `dotbot fw +fetch -f ` and flash those artifacts instead. + +## 1. Build and flash the DotBot + +The DotBot v3 is an nRF5340, which has two cores - the application core (your +app) and the network core (the radio) - so you build and flash two images: + +```bash +# build the bare dotbot apps into ./artifacts/ (needs SEGGER Embedded Studio) +dotbot fw artifacts --app dotbot +dotbot fw artifacts --app nrf5340_net --target nrf5340dk-net +# cable-flash to the bot whose J-Link serial starts with 77 +dotbot device flash dotbot -s 77 # app core +dotbot device flash nrf5340_net -b nrf5340dk-net -s 77 # network core +``` + +## 2. Build and flash the gateway + +The gateway is a dev board (e.g. an nRF52840-DK) plugged into your computer; it +bridges the robot's radio to USB serial. + +```bash +dotbot fw artifacts --app dotbot_gateway --target nrf52840dk +# cable-flash to the DK whose J-Link serial starts with 10 +dotbot device flash dotbot_gateway -b nrf52840dk -s 10 +``` + +## 3. Drive it from the web UI + +With the gateway plugged in, point the controller at its serial port and open +the web UI: + +```bash +dotbot run controller --conn /dev/ttyACM0 -w # serial gateway; no swarm-id needed +``` + +Select the bot in the browser and steer it with the joystick. See the +[controller + web UI guide](controller.md) for the full UI tour, and +[`fw`](../cli/fw.md) / [`device`](../cli/device.md) for the firmware commands. + +## Next + +- Operate many bots over the air - the [`swarm`](../cli/swarm.md) reference. +- Add real-world positions - [LH2 calibration](lh2-calibration.md). diff --git a/doc/index.md b/doc/index.md index 6c0f5b9b..8a8220a5 100644 --- a/doc/index.md +++ b/doc/index.md @@ -18,9 +18,7 @@ own code - one bot, or a swarm of hundreds. Pick a starting point: or gateway needed: `dotbot run simulator -w`. Then explore the [web-UI guide](guides/controller.md). - **Get one bot moving** - build and cable-flash a single DotBot and gateway, - then drive it from the browser. See the one-bot quickstart below - ([`fw`](cli/fw.md) / [`device`](cli/device.md) / - [controller guide](guides/controller.md)). + then drive it from the browser. See the [one-bot guide](guides/one-bot.md). - **Run a swarm experiment** - provision and command many bots over the air. The swarm quickstart below is the main path; then see [`swarm`](cli/swarm.md) and [LH2 localization](guides/lh2-calibration.md). From 09ede53d15f4eaf1b8550a6864afa16e9cce3887 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 06:55:47 +0200 Subject: [PATCH 02/27] readme: lead with the swarm quickstart, move one-bot and LH2 to guides The one-bot and LH2 quickstarts move into guides (doc/guides/one-bot.md and the existing LH2 guide), not removed - Going further links both, so the README leads straight into the swarm path. AI-assisted: Claude Opus 4.8 --- README.md | 66 +++++++------------------------------------------------ 1 file changed, 8 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index 20b1f2e0..5d772a7f 100644 --- a/README.md +++ b/README.md @@ -118,41 +118,6 @@ Commands: Every command and flag is documented in the [CLI reference][cli-doc]. -## Quickstart - one bot - -Build and flash firmware for a single dotbot: - -```bash -# build the bare dotbot apps into ./artifacts/ (needs SEGGER Embedded Studio) -# two steps because the DotBot has two cores -dotbot fw artifacts --app dotbot -dotbot fw artifacts --app nrf5340_net --target nrf5340dk-net -# cable-flash it to the bot whose J-Link serial starts with 77 -dotbot device flash dotbot -s 77 # app core -dotbot device flash nrf5340_net -b nrf5340dk-net -s 77 # network core -``` - -Now, build and flash the gateway to connect to a robot. -The gateway is a dev board (e.g. an nRF52840-DK) plugged into your -computer; it bridges the robot's radio to USB serial. - -```bash -# build the gateway firmware for your DK board into ./artifacts/ (needs SEGGER Embedded Studio) -dotbot fw artifacts --app dotbot_gateway --target nrf52840dk -# cable-flash it to the DK whose J-Link serial starts with 10 -dotbot device flash dotbot_gateway -b nrf52840dk -s 10 -``` - -With a gateway plugged into your computer, point the controller at it -and open the web UI: - -```bash -dotbot run controller --conn /dev/ttyACM0 -w # serial gateway; no swarm-id needed -``` - -More detail: building and flashing one board ([`fw`][fw-doc] / [`device`][device-doc]) -and driving it from the web UI ([controller guide][controller-doc]). - ## Quickstart - a swarm ### setup the swarm @@ -214,31 +179,15 @@ dotbot run controller -w # will open a webpage at http://localhost:8000/PyDotBo Full walkthrough of fleet operations - status, OTA flash, start/stop, monitor - is in the [`swarm` reference][swarm-doc]. -## Quickstart - Lighthouse 2 localization - -Give your robots a real-world `(x, y)` position. You'll need at least one -Lighthouse 2 base station and the calibration extra -(`pip install --pre 'pydotbot[calibrate]'`). - -```bash -# 1. flash the capture firmware to a cabled dotbot and collect four corner points -dotbot device flash lh2_calibration -s 77 -dotbot run lh2-calibration collect -p /dev/tty.usbmodem0007745943981 -d 200 # square of side 20 cm - -# 2. push the resulting calibration to the fleet over the air -dotbot swarm stop # ensure all robots are in bootloader -dotbot swarm calibrate-lh2 ~/.dotbot/calibration-2026-05-26T14-00-36Z.toml -``` - -Your bots now report their `(x, y)` location. The full setup - arena sizing, -base-station placement, and troubleshooting - is in the -[LH2 calibration guide][lh2-doc]. - ## Going further -Full command reference and guides - running the controller + web UI, the four -CLI namespaces (`fw` / `device` / `swarm` / `run`), hardware, and LH2 -calibration - are in the [documentation][doc-link]. +- **Drive a single bot** - build, flash, and control one DotBot end to end: + the [one-bot guide][one-bot-doc]. +- **Lighthouse 2 localization** - give your bots real-world `(x, y)` positions: + the [LH2 calibration guide][lh2-doc]. +- **Everything else** - the `dotbot` commands (`fw` / `device` / `swarm` / `run`, + plus `config`), the controller + web UI, and hardware notes are in the + [documentation][doc-link]. Swarm orchestration is in the base install. Only LH2 calibration needs an extra: @@ -278,6 +227,7 @@ See `LICENSE` in each component repository. [swarm-doc]: https://pydotbot.readthedocs.io/en/latest/cli/swarm.html [config-doc]: https://pydotbot.readthedocs.io/en/latest/reference/configuration.html [controller-doc]: https://pydotbot.readthedocs.io/en/latest/guides/controller.html +[one-bot-doc]: https://pydotbot.readthedocs.io/en/latest/guides/one-bot.html [lh2-doc]: https://pydotbot.readthedocs.io/en/latest/guides/lh2-calibration.html [troubleshooting-doc]: https://pydotbot.readthedocs.io/en/latest/reference/troubleshooting.html [rest-doc]: https://pydotbot.readthedocs.io/en/latest/reference/rest.html From 1f899555716e9bb8b16994c316e5bbb9d58caa02 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 07:43:42 +0200 Subject: [PATCH 03/27] dotbot: fetch and flash default to the latest swarmit release latest resolves via the GitHub /releases API (not /releases/latest, which skips prereleases like 0.8.0rcN). Resolution happens in the CLI layer before flash_role, so the concrete tag - not the literal "latest" - reaches resolve_fw_root and both the explicit fetch and the auto-fetch hook land in the same ./artifacts// dir. AI-assisted: Claude Opus 4.8 --- dotbot/cli/device.py | 24 ++++++++++++---- dotbot/cli/fw.py | 16 +++++++++-- dotbot/firmware/flash.py | 29 +++++++++++++++++++ dotbot/tests/test_device.py | 57 +++++++++++++++++++++++++++++-------- 4 files changed, 106 insertions(+), 20 deletions(-) diff --git a/dotbot/cli/device.py b/dotbot/cli/device.py index 954a5d11..9ff48a15 100644 --- a/dotbot/cli/device.py +++ b/dotbot/cli/device.py @@ -96,10 +96,10 @@ def _fw_version_option(f): return click.option( "--fw-version", "-f", - required=True, + default=None, help=( - "Release version to flash, e.g. 0.8.0rc1. Its binaries are " - "fetched into ./artifacts/ if not already cached." + "Release version to flash, e.g. 0.8.0rc2 (default: latest swarmit " + "release). Binaries are fetched into ./artifacts/ if not cached." ), )(f) @@ -135,7 +135,11 @@ def flash_swarmit_sandbox( network identity. Auto-fetches the release if not already in ./artifacts//. """ - from dotbot.firmware.flash import flash_role, normalize_network_id + from dotbot.firmware.flash import ( + flash_role, + normalize_network_id, + resolve_latest_version, + ) swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) if swarm_id is None: @@ -143,6 +147,9 @@ def flash_swarmit_sandbox( "no swarm id: pass --swarm-id, or set swarm_id (or a " "deployment) in your config." ) + if fw_version is None: + fw_version = resolve_latest_version() + click.echo(f"[INFO] latest swarmit release: {fw_version}") ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( @@ -171,7 +178,11 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): identity. Auto-fetches the release if absent. (To run the host-side UART<->MQTT bridge instead, use `dotbot run gateway`.) """ - from dotbot.firmware.flash import flash_role, normalize_network_id + from dotbot.firmware.flash import ( + flash_role, + normalize_network_id, + resolve_latest_version, + ) swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) if swarm_id is None: @@ -179,6 +190,9 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): "no swarm id: pass --swarm-id, or set swarm_id (or a " "deployment) in your config." ) + if fw_version is None: + fw_version = resolve_latest_version() + click.echo(f"[INFO] latest swarmit release: {fw_version}") ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index df927cdf..be7c3d1c 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -263,7 +263,10 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos @cmd.command() @click.option( - "--fw-version", "-f", required=True, help="Release version tag, or 'local'." + "--fw-version", + "-f", + default=None, + help="Release version tag (default: latest swarmit release), or 'local'.", ) @click.option( "--local-root", @@ -271,9 +274,16 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos help="Root of a local DotBot-firmware/swarmit build (with --fw-version local).", ) def fetch(fw_version, local_root): - """Download a released firmware set into ./artifacts//.""" - from dotbot.firmware.flash import fetch_assets + """Download a released firmware set into ./artifacts//. + + With no --fw-version, fetches the latest swarmit release (prereleases + included); the resolved tag is printed and used as the cache directory. + """ + from dotbot.firmware.flash import fetch_assets, resolve_latest_version + if fw_version is None: + fw_version = resolve_latest_version() + click.echo(f"[INFO] latest swarmit release: {fw_version}") out = fetch_assets(fw_version, artifacts_dir(), local_root) echo_artifact_path(out, action="fetched into") diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 389e89cf..867d8ceb 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -55,6 +55,7 @@ LH2_MATRIX_BYTES = 3 * 3 * 4 # 3x3 int32 matrix LH2_MAX_HOMOGRAPHIES = 16 RELEASE_BASE_URL = "https://github.com/DotBots/swarmit/releases/download" +RELEASE_API_URL = "https://api.github.com/repos/DotBots/swarmit/releases" # Application images are linked after the bootloader. APP_FLASH_BASE_ADDR = 0x00010000 # Programmer bring-up files @@ -140,6 +141,34 @@ def download_file(url: str, dest: Path) -> None: click.echo(f"[OK ] wrote {dest} ({len(data)} bytes)") +def resolve_latest_version() -> str: + """Resolve the newest swarmit release tag, prereleases included. + + Queries the GitHub releases API rather than ``/releases/latest``, which + excludes prereleases (the current newest, e.g. ``0.8.0rc2``, is one). + Unauthenticated; the 60 req/hour limit is fine for a CLI. + """ + url = f"{RELEASE_API_URL}?per_page=1" + click.echo(f"[GET ] {url}") + request = urllib.request.Request( + url, + headers={"Accept": "application/vnd.github+json", "User-Agent": "dotbot"}, + ) + try: + with urllib.request.urlopen(request) as resp: + releases = json.load(resp) + except (urllib.error.HTTPError, urllib.error.URLError) as exc: + raise click.ClickException( + f"Could not resolve the latest release ({exc}). " + "Pass an explicit version, e.g. -f 0.8.0rc2." + ) from exc + if not releases: + raise click.ClickException( + "No releases found for DotBots/swarmit; pass an explicit -f ." + ) + return releases[0]["tag_name"] + + def convert_bin_to_hex(bin_path: Path, base_addr: int) -> Path: if IntelHex is None: raise click.ClickException( diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index 08eda116..b54ccbf5 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -62,18 +62,14 @@ def test_flash_mari_gateway_rejects_calibration(runner): assert bad.exit_code != 0 -def test_flash_swarmit_sandbox_requires_swarm_id_and_version(runner): - """--swarm-id and -f are both required for flash-swarmit-sandbox.""" - assert ( - runner.invoke(device_cmd, ["flash-swarmit-sandbox", "-f", "0.8.0rc1"]).exit_code - != 0 - ) - assert ( - runner.invoke( - device_cmd, ["flash-swarmit-sandbox", "--swarm-id", "1234"] - ).exit_code - != 0 - ) +def test_flash_swarmit_sandbox_requires_swarm_id(runner): + """flash-swarmit-sandbox needs a swarm id (flag or config); -f is now + optional and defaults to the latest release (so no network in this test: + swarm_id is checked before the version resolves).""" + with runner.isolated_filesystem(): + result = runner.invoke(device_cmd, ["flash-swarmit-sandbox", "-f", "0.8.0rc1"]) + assert result.exit_code != 0 + assert "no swarm id" in result.output def test_flash_mari_gateway_help_disambiguates_from_bridge(runner): @@ -370,3 +366,40 @@ def fake_download(url, dest): monkeypatch.setattr(flash, "download_file", fake_download) with pytest.raises(click.ClickException): flash.fetch_assets("0.0.0-nope", tmp_path) + + +def test_resolve_latest_version_returns_newest_tag(monkeypatch): + """Returns the first (newest, prereleases included) tag from the API.""" + import io + import json + + import dotbot.firmware.flash as flash + + payload = json.dumps([{"tag_name": "0.8.0rc2"}, {"tag_name": "0.8.0rc1"}]).encode() + monkeypatch.setattr( + flash.urllib.request, "urlopen", lambda req: io.BytesIO(payload) + ) + assert flash.resolve_latest_version() == "0.8.0rc2" + + +def test_resolve_latest_version_no_releases_errors(monkeypatch): + """An empty release list is a clear error, not an IndexError.""" + import io + + import dotbot.firmware.flash as flash + + monkeypatch.setattr(flash.urllib.request, "urlopen", lambda req: io.BytesIO(b"[]")) + with pytest.raises(click.ClickException): + flash.resolve_latest_version() + + +def test_resolve_latest_version_network_error_errors(monkeypatch): + """A network failure surfaces as a friendly ClickException.""" + import dotbot.firmware.flash as flash + + def boom(req): + raise flash.urllib.error.URLError("offline") + + monkeypatch.setattr(flash.urllib.request, "urlopen", boom) + with pytest.raises(click.ClickException): + flash.resolve_latest_version() From eab623934ae45f784136ed76eadce02644bc3900 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 07:43:42 +0200 Subject: [PATCH 04/27] doc: fetch defaults to the latest release AI-assisted: Claude Opus 4.8 --- doc/cli/fw.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/cli/fw.md b/doc/cli/fw.md index 6fe4570b..5be7f63e 100644 --- a/doc/cli/fw.md +++ b/doc/cli/fw.md @@ -44,7 +44,7 @@ segger_dir = "/path/to/SEGGER Embedded Studio X.YY" |---|---| | Compile an app, leave it in the SES `Output/` tree (path echoed) | `dotbot fw build` | | Compile **and** collect a flat `-.hex` into `./artifacts/` | `dotbot fw artifacts` | -| Download a published release into `./artifacts//` | `dotbot fw fetch -f ` | +| Download a published release (latest by default) into `./artifacts//` | `dotbot fw fetch [-f ]` | | List targets you can build | `dotbot fw targets [--sandbox]` | | List what's cached locally | `dotbot fw list` | | A Makefile knob the CLI doesn't model | `dotbot fw make ` | @@ -124,7 +124,8 @@ dotbot fw artifacts --app nrf5340_net -t nrf5340dk-net dotbot fw artifacts --app spin -t dotbot-v3 --sandbox # Pull a published release instead of building → ./artifacts// -dotbot fw fetch -f v1.0.0 +dotbot fw fetch # latest swarmit release (prereleases included) +dotbot fw fetch -f 0.8.0rc2 # or pin an explicit version # See what's cached dotbot fw list From 774dad1f058165abf96bc355d5545ac31f98def2 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 07:43:42 +0200 Subject: [PATCH 05/27] readme: drop the hardcoded firmware version (fetch defaults to latest) AI-assisted: Claude Opus 4.8 --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 5d772a7f..a7ef3fd9 100644 --- a/README.md +++ b/README.md @@ -138,9 +138,9 @@ We also need a more powerful gateway firmware. Let's flash both - the network id comes from your config: ```bash -dotbot fw fetch -f 0.8.0rc1 # pull the pre-compiled firmwares from a release -dotbot device flash-mari-gateway -s 10 -f 0.8.0rc1 # flash the gateway -dotbot device flash-swarmit-sandbox -s 77 -f 0.8.0rc1 # the sandbox firmware - do this on each dotbot +dotbot fw fetch # pull the latest pre-compiled firmwares from a release +dotbot device flash-mari-gateway -s 10 # flash the gateway +dotbot device flash-swarmit-sandbox -s 77 # the sandbox firmware - do this on each dotbot ``` (`device flash-mari-gateway` / `flash-swarmit-sandbox` auto-fetch From 2b4ad879b19e7772eae90a1a5ad779b51d2b43f2 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 07:59:38 +0200 Subject: [PATCH 06/27] dotbot: clean up fetch output (package-manager style) AI-assisted: Claude Opus 4.8 --- dotbot/cli/device.py | 4 ++-- dotbot/cli/fw.py | 5 ++--- dotbot/firmware/flash.py | 45 ++++++++++++++++++++++++------------- dotbot/tests/test_device.py | 5 +++-- 4 files changed, 37 insertions(+), 22 deletions(-) diff --git a/dotbot/cli/device.py b/dotbot/cli/device.py index 9ff48a15..774645b3 100644 --- a/dotbot/cli/device.py +++ b/dotbot/cli/device.py @@ -149,7 +149,7 @@ def flash_swarmit_sandbox( ) if fw_version is None: fw_version = resolve_latest_version() - click.echo(f"[INFO] latest swarmit release: {fw_version}") + click.echo(f"No version specified, using the latest release: {fw_version}") ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( @@ -192,7 +192,7 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): ) if fw_version is None: fw_version = resolve_latest_version() - click.echo(f"[INFO] latest swarmit release: {fw_version}") + click.echo(f"No version specified, using the latest release: {fw_version}") ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index be7c3d1c..1c3248f6 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -283,9 +283,8 @@ def fetch(fw_version, local_root): if fw_version is None: fw_version = resolve_latest_version() - click.echo(f"[INFO] latest swarmit release: {fw_version}") - out = fetch_assets(fw_version, artifacts_dir(), local_root) - echo_artifact_path(out, action="fetched into") + click.echo(f"No version specified, fetching the latest release: {fw_version}") + fetch_assets(fw_version, artifacts_dir(), local_root) @cmd.command(name="list") diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 867d8ceb..2a0ef634 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -120,8 +120,22 @@ def resolve_fw_root(bin_dir: Path, fw_version: str) -> Path: return bin_dir / fw_version -def download_file(url: str, dest: Path) -> None: - click.echo(f"[GET ] {url}") +def _human_size(num_bytes: int) -> str: + size = float(num_bytes) + for unit in ("B", "KB", "MB", "GB"): + if size < 1024 or unit == "GB": + return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}" + size /= 1024 + + +def _short_path(path: Path) -> str: + """Path relative to the cwd when that's shorter, else absolute.""" + rel = os.path.relpath(path) + return rel if not rel.startswith("..") else str(path) + + +def download_file(url: str, dest: Path) -> int: + """Download ``url`` to ``dest``; return the number of bytes written.""" try: with urllib.request.urlopen(url) as resp: status = getattr(resp, "status", 200) @@ -138,7 +152,7 @@ def download_file(url: str, dest: Path) -> None: ) from exc dest.write_bytes(data) - click.echo(f"[OK ] wrote {dest} ({len(data)} bytes)") + return len(data) def resolve_latest_version() -> str: @@ -149,7 +163,6 @@ def resolve_latest_version() -> str: Unauthenticated; the 60 req/hour limit is fine for a CLI. """ url = f"{RELEASE_API_URL}?per_page=1" - click.echo(f"[GET ] {url}") request = urllib.request.Request( url, headers={"Accept": "application/vnd.github+json", "User-Agent": "dotbot"}, @@ -375,7 +388,6 @@ def fetch_assets( out_dir = resolve_fw_root(bin_dir, fw_version) out_dir.mkdir(parents=True, exist_ok=True) - click.echo(f"[INFO] target dir: {out_dir.resolve()}") if fw_version == "local": local_root = local_root.expanduser().resolve() @@ -424,21 +436,24 @@ def fetch_assets( "move-dotbot-v3.bin", "motors-dotbot-v3.bin", ] + click.echo(f"Fetching {fw_version} into {_short_path(out_dir)}") for name in assets: url = f"{RELEASE_BASE_URL}/{fw_version}/{name}" - dest = out_dir / name - download_file(url, dest) + size = download_file(url, out_dir / name) + click.echo(f" {name} ({_human_size(size)})") + skipped = 0 for name in example_bins: url = f"{RELEASE_BASE_URL}/{fw_version}/{name}" - dest = out_dir / name try: - download_file(url, dest) - except click.ClickException as exc: - click.echo( - f"[skip] optional sample app {name} not in release " - f"{fw_version} ({exc.format_message()})", - err=True, - ) + size = download_file(url, out_dir / name) + except click.ClickException: + skipped += 1 + continue + click.echo(f" {name} ({_human_size(size)})") + if skipped: + click.echo(f" ({skipped} optional sample app(s) not in this release)") + count = len(assets) + len(example_bins) - skipped + click.echo(f"Done: {count} file(s) in {_short_path(out_dir)}") return out_dir diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index b54ccbf5..26ca75e6 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -346,8 +346,9 @@ def fake_download(url, dest): if name.endswith(".hex"): # the 4 required system images dest.write_bytes(b"\x00") downloaded.append(name) - else: # optional sample .bin → simulate a release 404 - raise click.ClickException(f"HTTP Error 404: {name}") + return 1 # bytes written (download_file returns the size) + # optional sample .bin → simulate a release 404 + raise click.ClickException(f"HTTP Error 404: {name}") monkeypatch.setattr(flash, "download_file", fake_download) out = flash.fetch_assets("0.8.0rc1", tmp_path) # must not raise From 0a96e95e26b80dd7e345517a0808181ee2c2c5d2 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 09:13:11 +0200 Subject: [PATCH 07/27] dotbot: add a bare-REST 'circle' demo (dotbot run demo circle) Drives the first bot in a circle over plain HTTP. Adds `requests` as a dep: the demo (and the docs scripting example) talk to the controller with no async and no internal client, so it doubles as a copy-paste template; the internal httpx-based client stays for the richer examples. AI-assisted: Claude Opus 4.8 --- dotbot/cli/demo.py | 2 + dotbot/examples/circle/__init__.py | 2 + dotbot/examples/circle/circle.py | 61 ++++++++++++++++++++++++++++++ pyproject.toml | 1 + 4 files changed, 66 insertions(+) create mode 100644 dotbot/examples/circle/__init__.py create mode 100644 dotbot/examples/circle/circle.py diff --git a/dotbot/cli/demo.py b/dotbot/cli/demo.py index 940bdeee..ff78543c 100644 --- a/dotbot/cli/demo.py +++ b/dotbot/cli/demo.py @@ -12,6 +12,7 @@ import click +from dotbot.examples.circle.circle import main as _circle_main from dotbot.examples.qrkey_demo.cli import main as _qrkey_main @@ -43,4 +44,5 @@ def cmd(ctx, list_demos): # We pass `name=...` rather than mutating `_qrkey_main.name` so the # demo's own test suite (which imports the same Click command) stays # unaffected. +cmd.add_command(_circle_main, name="circle") cmd.add_command(_qrkey_main, name="qr") diff --git a/dotbot/examples/circle/__init__.py b/dotbot/examples/circle/__init__.py new file mode 100644 index 00000000..2eb0e0b9 --- /dev/null +++ b/dotbot/examples/circle/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: 2026-present Inria +# SPDX-License-Identifier: BSD-3-Clause diff --git a/dotbot/examples/circle/circle.py b/dotbot/examples/circle/circle.py new file mode 100644 index 00000000..0cd53ca8 --- /dev/null +++ b/dotbot/examples/circle/circle.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: 2026-present Inria +# SPDX-License-Identifier: BSD-3-Clause + +"""The simplest DotBot demo: drive one bot in a circle over the REST API. + +No pydotbot internals - just HTTP with `requests`, so it doubles as a +copy-pasteable template for your own scripts. Start a controller +(e.g. `dotbot run simulator -w`), then run `dotbot run demo circle`. +""" + +import time + +import click +import requests + + +@click.command(name="circle", help="Drive one bot in a circle (bare-REST demo).") +@click.option( + "--base", + default="http://localhost:8000", + show_default=True, + help="Controller REST base URL.", +) +@click.option( + "--seconds", + "-t", + default=5.0, + show_default=True, + help="How long to drive, in seconds.", +) +def main(base, seconds): + """Drive the first DotBot the controller sees in a circle, then stop.""" + try: + bots = requests.get(f"{base}/controller/dotbots", timeout=5).json() + except requests.RequestException as exc: + raise click.ClickException( + f"Cannot reach a controller at {base} ({exc}). " + "Start one first, e.g. `dotbot run simulator -w`." + ) from exc + if not bots: + raise click.ClickException( + f"No DotBots at {base}. The simulator (`dotbot run simulator -w`) " + "spawns one; on hardware, flash and connect a bot first." + ) + + address = bots[0]["address"] + move = f"{base}/controller/dotbots/{address}/0/move_raw" + click.echo(f"Driving {address} in a circle for {seconds:.0f}s (Ctrl-C to stop)...") + + # left_y and right_y are the two wheel speeds; unequal -> the bot turns. + end = time.time() + seconds + try: + while time.time() < end: + requests.put( + move, json={"left_x": 0, "left_y": 60, "right_x": 0, "right_y": 30} + ) + time.sleep(0.1) + finally: + # always stop the motors, even on Ctrl-C + requests.put(move, json={"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}) + click.echo("Done.") diff --git a/pyproject.toml b/pyproject.toml index 329202dc..4c3e53f5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,6 +42,7 @@ dependencies = [ "pynput >= 1.8.1", "pyserial >= 3.5", "qrkey >= 0.12.2", + "requests >= 2.31.0", "rich >= 14.0.0", "structlog >= 24.4.0", "uvicorn >= 0.32.0", From 6e0b93def8f0cd942431ae484cbbe960c7f70c91 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 09:13:11 +0200 Subject: [PATCH 08/27] doc: add a simulator guide AI-assisted: Claude Opus 4.8 --- doc/guides/index.md | 3 ++ doc/guides/simulator.md | 66 +++++++++++++++++++++++++++++++++++++++++ doc/index.md | 4 +-- 3 files changed, 71 insertions(+), 2 deletions(-) create mode 100644 doc/guides/simulator.md diff --git a/doc/guides/index.md b/doc/guides/index.md index cb5c1e9a..cf8085d4 100644 --- a/doc/guides/index.md +++ b/doc/guides/index.md @@ -4,11 +4,14 @@ Task-oriented walkthroughs that span several commands. ```{toctree} :hidden: +simulator one-bot controller lh2-calibration ``` +- [Try it in the simulator](simulator.md) - run the full UI and script bots with + no hardware. - [Drive a single DotBot](one-bot.md) - build, flash, and control one bot end to end. - [Run the controller + web UI](controller.md) - drive and visualize a swarm diff --git a/doc/guides/simulator.md b/doc/guides/simulator.md new file mode 100644 index 00000000..75b1e717 --- /dev/null +++ b/doc/guides/simulator.md @@ -0,0 +1,66 @@ +# Try it in the simulator + +The simulator runs the **full controller and web UI with no hardware** - no bot, +no gateway, no radio. It's the fastest way to see DotBot work, and because it +exposes the exact same REST/WebSocket API as the real controller, code you write +against the simulator runs unchanged against real robots. + +## Start it + +```bash +dotbot run simulator -w +``` + +This opens the web UI at driving a simulated +swarm. `dotbot run simulator` is shorthand for +`dotbot run controller --conn simulator`, so everything in the +[controller + web UI guide](controller.md) applies. Drive the bots from the +joystick and watch them on the map. + +## Drive one bot in a circle + +The simplest demo - it grabs the first bot the controller sees and drives it in a +circle. With the simulator running, in a second terminal: + +```bash +dotbot run demo circle +``` + +## Drive it from your own code + +That demo is ~15 lines talking to the controller's REST API over +[`requests`](https://pypi.org/project/requests/) (bundled with pydotbot) - here +is the whole thing, a template for your own scripts: + +```python +import requests, time + +BASE = "http://localhost:8000" +bot = requests.get(f"{BASE}/controller/dotbots").json()[0]["address"] + +# roll in a circle for ~5 s - left_y and right_y are the two wheel speeds +for _ in range(50): + requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", + json={"left_x": 0, "left_y": 60, "right_x": 0, "right_y": 30}) + time.sleep(0.1) +requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", + json={"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}) +``` + +The full surface - every endpoint, the live WebSocket stream, and CSV data +logging - is in the [REST / WebSocket reference](../reference/rest.md) (or the +[MQTT bridge](../reference/mqtt.md)). A higher-level Python SDK is planned; today +you talk to the controller over REST/WebSocket/MQTT. + +## More examples + +```bash +dotbot run demo --list # what's available +dotbot run demo qr # phone-as-joystick over QrKey +``` + +Richer multi-bot scenarios - work-and-charge, charging-station, labyrinth, the +naming game, motion shapes - live in `dotbot/examples/`, each with its own +README and a simulator init state. They drive the controller over the same +REST/WebSocket API shown above, so they run against the simulator or real +hardware unchanged. diff --git a/doc/index.md b/doc/index.md index 8a8220a5..fdcce713 100644 --- a/doc/index.md +++ b/doc/index.md @@ -15,8 +15,8 @@ New here? DotBots are small wheeled robots you drive from your browser or your own code - one bot, or a swarm of hundreds. Pick a starting point: - **Try it with no hardware** - the simulator runs the full web UI with no bot - or gateway needed: `dotbot run simulator -w`. Then explore the - [web-UI guide](guides/controller.md). + or gateway needed: `dotbot run simulator -w`. See the + [simulator guide](guides/simulator.md). - **Get one bot moving** - build and cable-flash a single DotBot and gateway, then drive it from the browser. See the [one-bot guide](guides/one-bot.md). - **Run a swarm experiment** - provision and command many bots over the air. From 40d6a47b50744d646a49f8acb8814e96907aef46 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 09:13:11 +0200 Subject: [PATCH 09/27] readme: restructure onboarding around install, simulator, and swarm The scripting example moved into the simulator guide (not removed); the page now leads with install, then the no-hardware simulator, then the "Deploy a real swarm" path. AI-assisted: Claude Opus 4.8 --- README.md | 111 ++++++++++++++++++------------------------------------ 1 file changed, 37 insertions(+), 74 deletions(-) diff --git a/README.md b/README.md index a7ef3fd9..2d8bf98c 100644 --- a/README.md +++ b/README.md @@ -6,19 +6,13 @@ # PyDotBot -**The control plane for [DotBot](http://www.dotbots.org) swarms - build firmware, -flash a robot, and control a fleet over the air, from one bot to a thousand, all -from a single `dotbot` CLI and web UI.** +The control plane for the [DotBot](http://www.dotbots.org) - a small wireless +wheeled robot built to operate in large swarms, for research and education. -DotBots are small wireless wheeled robots built to operate in large swarms, -for research and education. Developed by the [AIO team](https://aio.inria.fr/) at -[Inria Paris](https://www.inria.fr/), and run routinely with ~100–200 bots, -with one 725-bot campaign. +PyDotBot allows you to flash a robot and control a whole fleet over the air, +from one bot to a thousand. -▶️ [Click to see a DotBot swarm in action](https://www.youtube.com/watch?v=pXGTLqafReU) - -PyDotBot is the control plane in the middle: your code, the web UI, and users -talk to it, and it drives the swarm through a gateway. +[▶️ Click to see a DotBot swarm in action](https://www.youtube.com/watch?v=pXGTLqafReU) ```text ┌───────────┐ ┌────────────┐ ┌─────────┐ @@ -37,90 +31,53 @@ talk to it, and it drives the swarm through a gateway. - 🧪 Try it all with **zero hardware** using the built-in simulator - 🛠️ One `dotbot` CLI takes you from build → flash → run -## Try it now - no hardware +## Install -See the whole thing run with nothing but Python: +PyDotBot is available on [PyPi](https://pypi.org/project/pydotbot/), install it with: ```bash -pip install --pre pydotbot # using 'pre' while we are at release candidate -dotbot run simulator -w # opens the web UI at http://localhost:8000/PyDotBot/, driving a simulated swarm +pip install --pre pydotbot ``` -Drive the simulated bots from the joystick + map - then script them from your own -code (below), or set up real hardware further down. +Then, check your installation with `dotbot --version` and learn what's possible with `dotbot --help`. -## Drive it from your own code +Every command and flag is documented in the [CLI reference][cli-doc]. -The controller - real or simulated - exposes a REST + WebSocket API, so you can -command the swarm in a few lines of Python (only extra dependency: -[`requests`](https://pypi.org/project/requests/)): +## Try the simulator -```python -import requests, time +See the whole thing run with nothing but Python! -BASE = "http://localhost:8000" -bot = requests.get(f"{BASE}/controller/dotbots").json()[0]["address"] +The command below will run a simulated swarm, which you can observe in a web UI at http://localhost:8000/PyDotBot/ : -# roll in a circle for ~5 s - left_y and right_y are the two wheel speeds -for _ in range(50): - requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", - json={"left_x": 0, "left_y": 60, "right_x": 0, "right_y": 30}) - time.sleep(0.1) -requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", - json={"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}) +```bash +dotbot run simulator -w ``` -The full surface - every endpoint, the live WebSocket stream, and CSV data -logging - is in the [REST / WebSocket reference][rest-doc] (or the -[MQTT bridge][mqtt-doc]). A higher-level Python SDK is planned; today you talk to -the controller over REST/WebSocket/MQTT. +Drive the simulated bots from the UI, or run a bundled demo in a +second terminal: -The firmware for the DotBots can be found [here][dotbot-firmware-repo]. +```bash +dotbot run demo circle # drive one bot in a circle (the simplest demo) +``` -## Prerequisites (for real hardware) +Learn how to script the swarm from your own code, run the richer examples, and more - all with +no hardware - in the [simulator guide][simulator-doc]. -Driving an already-provisioned swarm - or the simulator above - needs nothing but -Python. The tools below are only for building or cable-flashing firmware yourself. +## Deploy a real swarm -Software to install (as needed): -- Python ≥ 3.11 - ensure you also have [pip](https://pip.pypa.io/en/stable/) available in your PATH -- [nRF Command Line Tools](https://www.nordicsemi.com/Products/Development-tools/nRF-Command-Line-Tools) (`nrfjprog`), for commands such as `dotbot device flash` -- [SEGGER Embedded Studio](https://www.segger.com/products/development-tools/embedded-studio/), for commands such as `dotbot fw build` +The DotBot is made to operate as a swarm, here is how you can deploy it on real robots. + +### Prerequisites Minimal hardware setup: - DotBot v3, as well as a USB-C cable and a barrel-jack charger (2.5 mm, 6–18 V, 5/10 A) - nRF5340-DK to use as gateway, as well as a micro-USB cable -## Install - -```bash -pip install --pre pydotbot # --pre while 0.29 is in pre-release -git clone --recurse-submodules --branch develop https://github.com/DotBots/DotBot-firmware.git -``` - -## Usage - -``` -$ dotbot --help -Usage: dotbot [OPTIONS] COMMAND [ARGS]... - - One CLI for the whole DotBot workflow: build and flash firmware, program and - control a single robot, and run experiments over the air across a swarm - - from one bot to a thousand. - -Commands: - fw Firmware artifacts (no hardware): build / fetch / list / make. - device One connected device (cable/probe): flash an app/role, read info. - swarm The fleet over the air: status, start/stop, OTA flash, monitor. - run Host-side processes: controller, gateway, simulator, calibration, demos, teleop. - config Show the resolved config and where it came from; scaffold one with init. -``` - -Every command and flag is documented in the [CLI reference][cli-doc]. - -## Quickstart - a swarm +Software to install (as needed): +- Python ≥ 3.11 - ensure you also have [pip](https://pip.pypa.io/en/stable/) available in your PATH +- [nRF Command Line Tools](https://www.nordicsemi.com/Products/Development-tools/nRF-Command-Line-Tools) (`nrfjprog`), for commands such as `dotbot device flash` -### setup the swarm +### Setup To operate as a swarm, set your swarm connection config: @@ -152,7 +109,7 @@ Now, run the gateway (the broker comes from your config): dotbot run gateway -p /dev/cu.usbmodem0010500324491 ``` -### use the swarm +### Deploy and control You can flash as many dotbots as you want, all at once! First, how about making them spinnnn 🔄 🔄 @@ -181,6 +138,11 @@ is in the [`swarm` reference][swarm-doc]. ## Going further +- **Download the firmware repository**: +```bash +git clone --recurse-submodules --branch develop https://github.com/DotBots/DotBot-firmware.git +``` +- Install [SEGGER Embedded Studio](https://www.segger.com/products/development-tools/embedded-studio/), for commands such as `dotbot fw build` - **Drive a single bot** - build, flash, and control one DotBot end to end: the [one-bot guide][one-bot-doc]. - **Lighthouse 2 localization** - give your bots real-world `(x, y)` positions: @@ -226,6 +188,7 @@ See `LICENSE` in each component repository. [device-doc]: https://pydotbot.readthedocs.io/en/latest/cli/device.html [swarm-doc]: https://pydotbot.readthedocs.io/en/latest/cli/swarm.html [config-doc]: https://pydotbot.readthedocs.io/en/latest/reference/configuration.html +[simulator-doc]: https://pydotbot.readthedocs.io/en/latest/guides/simulator.html [controller-doc]: https://pydotbot.readthedocs.io/en/latest/guides/controller.html [one-bot-doc]: https://pydotbot.readthedocs.io/en/latest/guides/one-bot.html [lh2-doc]: https://pydotbot.readthedocs.io/en/latest/guides/lh2-calibration.html From 859cd8d6f20ee770e481d87faea5137fa2e5ccac Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 11:55:30 +0200 Subject: [PATCH 10/27] dotbot: multi-source firmware fetch + ~/.dotbot/artifacts cache layout The cache moves to user-level ~/.dotbot/artifacts/ (shared across dirs; override DOTBOT_ARTIFACTS_DIR), as -/ dirs + a manifest, so swarmit and DotBot-firmware (independent version schemes) never collide. `fw fetch` pulls from both sources, downloading every .hex/.bin a release lists via the GitHub API (fixing the silent .bin 404s). `device flash ` searches the cache, preferring a local build over the newest release. AI-assisted: Claude Opus 4.8 --- dotbot/cli/_artifacts.py | 51 +++++++-- dotbot/cli/fw.py | 50 +++++--- dotbot/firmware/flash.py | 221 +++++++++++++++++++++--------------- dotbot/tests/test_device.py | 47 ++++---- 4 files changed, 233 insertions(+), 136 deletions(-) diff --git a/dotbot/cli/_artifacts.py b/dotbot/cli/_artifacts.py index dab10222..9b87c681 100644 --- a/dotbot/cli/_artifacts.py +++ b/dotbot/cli/_artifacts.py @@ -15,19 +15,22 @@ SES nor a firmware repo) stays cheap and side-effect-free. """ +import os from pathlib import Path import click def artifacts_dir() -> Path: - """The CWD-local ``./artifacts/`` directory, resolved absolute. + """The firmware cache: ``~/.dotbot/artifacts/`` by default. - The single source of truth for where build outputs land and fetched - releases are cached. Per-workspace (no global ``~/.dotbot/fw``), so - two checkouts never collide. + User-level and shared across working directories (like other tools cache + downloaded firmware), so you don't re-download a release per directory and + the launch dir stays clean. Override with ``$DOTBOT_ARTIFACTS_DIR``. """ - return (Path.cwd() / "artifacts").resolve() + override = os.environ.get("DOTBOT_ARTIFACTS_DIR") + base = Path(override) if override else Path.home() / ".dotbot" / "artifacts" + return base.expanduser().resolve() def echo_artifact_path(path: Path, *, action: str = "using") -> None: @@ -59,6 +62,36 @@ def ensure_nrfjprog() -> None: raise friendly_nrfjprog_error() +def _find_in_cache(name: str) -> Path | None: + """Find a firmware file across the source-qualified cache dirs. + + The cache holds ``-/`` (fetched releases) and + ``-local[-link]/`` (built/linked) subdirs. Prefer a local build + (you're iterating on it), else the newest-versioned release dir. + """ + root = artifacts_dir() + if not root.is_dir(): + return None + local_hits: list[Path] = [] + released_hits: list[Path] = [] + for sub in sorted(root.iterdir()): + if not sub.is_dir(): + continue + candidate = sub / name + if candidate.is_file(): + if sub.name.endswith(("-local", "-local-link")): + local_hits.append(candidate) + else: + released_hits.append(candidate) + if local_hits: + return local_hits[0] + if released_hits: + # newest by dir name (a coarse version sort; good enough for picking + # the latest release of a given app) + return sorted(released_hits, key=lambda p: p.parent.name, reverse=True)[0] + return None + + def resolve_app_artifact( app: str, *, @@ -80,8 +113,8 @@ def resolve_app_artifact( - Else, a friendly error telling the user to build or fetch first. """ name = f"{app}-sandbox-{board}.bin" if sandbox else f"{app}-{board}.hex" - cached = artifacts_dir() / name - if cached.is_file(): + cached = _find_in_cache(name) + if cached is not None: echo_artifact_path(cached, action="using") return cached @@ -114,6 +147,6 @@ def resolve_app_artifact( "DotBot-firmware source to build from.\n" " • `dotbot fw build " f"-a {app} -t {board}{' --sandbox' if sandbox else ''}` to build, or\n" - " • `dotbot fw fetch -f ` to download a release, then retry, or\n" - " • pass an explicit path: `dotbot device flash ./artifacts/`." + " • `dotbot fw fetch` to download the latest releases, then retry, or\n" + " • pass an explicit path: `dotbot device flash `." ) diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index 1c3248f6..76a9fc7e 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -222,7 +222,12 @@ def list_targets(sandbox): @click.option("-v", "--verbose", is_flag=True, default=False) @click.pass_context def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbose): - """Build + collect artifacts into ./artifacts/ (default).""" + """Build + collect artifacts into the local cache (default). + + Without --out, built apps land in the source-qualified + ``/dotbot-firmware-local/`` dir so `device flash ` finds them + alongside fetched releases. + """ import shutil target = from_config(ctx, "target", "board", "fw") @@ -237,7 +242,11 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos ) click.echo(str(artifact_path(build_target, project, config))) return - out = Path(out_dir).resolve() if out_dir else artifacts_dir() + out = ( + Path(out_dir).resolve() + if out_dir + else artifacts_dir() / "dotbot-firmware-local" + ) click.echo( f"Building + collecting artifacts for {target} ({config}) → {out}/...", err=True, @@ -262,29 +271,44 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos @cmd.command() +@click.option( + "--source", + "-S", + type=click.Choice(list(("swarmit", "dotbot-firmware"))), + default=None, + help="Limit to one source (default: fetch the latest from all sources).", +) @click.option( "--fw-version", "-f", default=None, - help="Release version tag (default: latest swarmit release), or 'local'.", + help="Release tag for --source (default: latest), or 'local'.", ) @click.option( "--local-root", type=click.Path(path_type=Path, file_okay=False, dir_okay=True), - help="Root of a local DotBot-firmware/swarmit build (with --fw-version local).", + help="Root of a local build tree (with --source --fw-version local).", ) -def fetch(fw_version, local_root): - """Download a released firmware set into ./artifacts//. +def fetch(source, fw_version, local_root): + """Download released firmware into ~/.dotbot/artifacts/-/. - With no --fw-version, fetches the latest swarmit release (prereleases - included); the resolved tag is printed and used as the cache directory. + With no flags, fetches the latest release from every source: swarmit (the + swarm system images) and DotBot-firmware (bare + sandbox apps). The two + version independently, so pinning a version with -f requires a --source. """ - from dotbot.firmware.flash import fetch_assets, resolve_latest_version + from dotbot.firmware.flash import DEFAULT_FETCH_SOURCES, fetch_assets - if fw_version is None: - fw_version = resolve_latest_version() - click.echo(f"No version specified, fetching the latest release: {fw_version}") - fetch_assets(fw_version, artifacts_dir(), local_root) + if fw_version is not None and source is None: + raise click.ClickException( + "Pass --source with -f/--fw-version: swarmit and dotbot-firmware " + "version independently." + ) + sources = [source] if source else list(DEFAULT_FETCH_SOURCES) + for src in sources: + version = fw_version or "latest" + if version == "latest": + click.echo(f"Fetching the latest {src} release...") + fetch_assets(src, version, artifacts_dir(), local_root) @cmd.command(name="list") diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 2a0ef634..402d471f 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -17,6 +17,7 @@ import time import urllib.error import urllib.request +from datetime import datetime, timezone from pathlib import Path import click @@ -54,8 +55,16 @@ # dotbot-lh2-calibration (1-byte count + N matrices of 3x3 int32 LE). LH2_MATRIX_BYTES = 3 * 3 * 4 # 3x3 int32 matrix LH2_MAX_HOMOGRAPHIES = 16 -RELEASE_BASE_URL = "https://github.com/DotBots/swarmit/releases/download" -RELEASE_API_URL = "https://api.github.com/repos/DotBots/swarmit/releases" +GITHUB_API = "https://api.github.com/repos" +# Firmware release sources. swarmit ships the swarm system images (bootloader +# + mari netcore + mari gateway); DotBot-firmware ships the bare apps (.hex) +# and the sandbox apps (.bin). Each is cached in its own -/ +# subdir of the artifacts cache so versions and provenance never collide. +RELEASE_SOURCES = { + "swarmit": "DotBots/swarmit", + "dotbot-firmware": "DotBots/DotBot-firmware", +} +DEFAULT_FETCH_SOURCES = ("swarmit", "dotbot-firmware") # Application images are linked after the bootloader. APP_FLASH_BASE_ADDR = 0x00010000 # Programmer bring-up files @@ -116,8 +125,8 @@ def normalize_network_id(raw: str | None) -> tuple[int, str] | None: return value, f"{value:04X}" -def resolve_fw_root(bin_dir: Path, fw_version: str) -> Path: - return bin_dir / fw_version +def resolve_fw_root(bin_dir: Path, source: str, fw_version: str) -> Path: + return bin_dir / f"{source}-{fw_version}" def _human_size(num_bytes: int) -> str: @@ -155,31 +164,44 @@ def download_file(url: str, dest: Path) -> int: return len(data) -def resolve_latest_version() -> str: - """Resolve the newest swarmit release tag, prereleases included. - - Queries the GitHub releases API rather than ``/releases/latest``, which - excludes prereleases (the current newest, e.g. ``0.8.0rc2``, is one). - Unauthenticated; the 60 req/hour limit is fine for a CLI. - """ - url = f"{RELEASE_API_URL}?per_page=1" +def _github_get(url: str): + """Unauthenticated GitHub API GET (60 req/hour is plenty for a CLI).""" request = urllib.request.Request( url, headers={"Accept": "application/vnd.github+json", "User-Agent": "dotbot"}, ) try: with urllib.request.urlopen(request) as resp: - releases = json.load(resp) + return json.load(resp) except (urllib.error.HTTPError, urllib.error.URLError) as exc: + raise click.ClickException(f"GitHub API request failed ({url}): {exc}") from exc + + +def resolve_release(source: str, fw_version: str) -> dict: + """Return the GitHub release JSON for ``source`` at ``fw_version``. + + ``fw_version="latest"`` resolves the newest release via ``/releases`` + (prereleases included, unlike ``/releases/latest`` which skips them - the + current newest, e.g. swarmit 0.8.0rc2, is a prerelease). + """ + if source not in RELEASE_SOURCES: raise click.ClickException( - f"Could not resolve the latest release ({exc}). " - "Pass an explicit version, e.g. -f 0.8.0rc2." - ) from exc - if not releases: - raise click.ClickException( - "No releases found for DotBots/swarmit; pass an explicit -f ." + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." ) - return releases[0]["tag_name"] + repo = RELEASE_SOURCES[source] + if fw_version == "latest": + releases = _github_get(f"{GITHUB_API}/{repo}/releases?per_page=1") + if not releases: + raise click.ClickException( + f"No releases found for {repo}; pass an explicit -f ." + ) + return releases[0] + return _github_get(f"{GITHUB_API}/{repo}/releases/tags/{fw_version}") + + +def resolve_latest_version(source: str = "swarmit") -> str: + """The newest release tag for ``source`` (prereleases included).""" + return resolve_release(source, "latest")["tag_name"] def convert_bin_to_hex(bin_path: Path, base_addr: int) -> Path: @@ -369,15 +391,21 @@ def manifest_matches( def fetch_assets( - fw_version: str, bin_dir: Path, local_root: Path | None = None + source: str, fw_version: str, bin_dir: Path, local_root: Path | None = None ) -> Path: - """Download (or symlink, for --fw-version=local) the testbed firmware - assets into ``bin_dir//`` and return that directory. - - The single source of truth for "get the system firmware". Used by - `dotbot fw fetch` and by the auto-fetch hook in `flash_role` - (fetch-if-absent). + """Fetch one source's firmware into ``bin_dir/-/``. + + For a released version, downloads every ``.hex``/``.bin`` asset the GitHub + release publishes (so it adapts to whatever the release ships, no hardcoded + asset list) and writes a ``manifest.json`` with provenance. For + ``fw_version="local"``, symlinks/copies from a local build tree into + ``-local/``. Used by `dotbot fw fetch` and the auto-fetch hook in + `flash_role`. """ + if source not in RELEASE_SOURCES: + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) if fw_version == "local" and not local_root: raise click.ClickException("--local-root is required when --fw-version=local.") if fw_version != "local" and local_root: @@ -386,74 +414,81 @@ def fetch_assets( err=True, ) - out_dir = resolve_fw_root(bin_dir, fw_version) - out_dir.mkdir(parents=True, exist_ok=True) - if fw_version == "local": - local_root = local_root.expanduser().resolve() - mapping = { - "bootloader-dotbot-v3.hex": local_root - / "device/bootloader/Output/dotbot-v3/Debug/Exe/bootloader-dotbot-v3.hex", - "netcore-nrf5340-net.hex": local_root - / "device/network_core/Output/nrf5340-net/Debug/Exe/netcore-nrf5340-net.hex", - "03app_gateway_app-nrf5340-app.hex": local_root - / "mari/firmware/app/03app_gateway_app/Output/nrf5340-app/Debug/Exe/03app_gateway_app-nrf5340-app.hex", - "03app_gateway_net-nrf5340-net.hex": local_root - / "mari/firmware/app/03app_gateway_net/Output/nrf5340-net/Debug/Exe/03app_gateway_net-nrf5340-net.hex", - } - - missing = [name for name, src in mapping.items() if not src.exists()] - if missing: - missing_list = ", ".join(missing) - raise click.ClickException(f"Missing local build artifacts: {missing_list}") - - for name, src in mapping.items(): - dest = out_dir / name - if dest.exists() or dest.is_symlink(): - dest.unlink() - try: - os.symlink(src, dest) - click.echo(f"[LINK] {dest} -> {src}") - except OSError: - shutil.copy2(src, dest) - click.echo(f"[COPY] {dest} <- {src}") - return out_dir + return _link_local_assets(source, local_root, bin_dir) + release = resolve_release(source, fw_version) + tag = release["tag_name"] + out_dir = resolve_fw_root(bin_dir, source, tag) + out_dir.mkdir(parents=True, exist_ok=True) assets = [ - "bootloader-dotbot-v3.hex", - "netcore-nrf5340-net.hex", - "03app_gateway_app-nrf5340-app.hex", - "03app_gateway_net-nrf5340-net.hex", + a for a in release.get("assets", []) if a["name"].endswith((".hex", ".bin")) ] - # Optional sample sandbox apps. These are built from DotBot-firmware's - # apps-sandbox/ and aren't guaranteed to be on every swarmit release, so - # a 404 here is expected, not fatal — the four system images above are - # all that provisioning (flash-swarmit-sandbox / flash-mari-gateway) needs. - example_bins = [ - "dotbot-dotbot-v3.bin", - "spin-dotbot-v3.bin", - "rgbled-dotbot-v3.bin", - "move-dotbot-v3.bin", - "motors-dotbot-v3.bin", - ] - click.echo(f"Fetching {fw_version} into {_short_path(out_dir)}") - for name in assets: - url = f"{RELEASE_BASE_URL}/{fw_version}/{name}" - size = download_file(url, out_dir / name) - click.echo(f" {name} ({_human_size(size)})") - skipped = 0 - for name in example_bins: - url = f"{RELEASE_BASE_URL}/{fw_version}/{name}" + if not assets: + raise click.ClickException( + f"{source} release {tag} publishes no .hex/.bin assets." + ) + click.echo(f"Fetching {source} {tag} into {_short_path(out_dir)}") + names = [] + for asset in assets: + size = download_file(asset["browser_download_url"], out_dir / asset["name"]) + click.echo(f" {asset['name']} ({_human_size(size)})") + names.append(asset["name"]) + _write_manifest(out_dir, source, tag, RELEASE_SOURCES[source], names) + click.echo(f"Done: {len(names)} file(s) in {_short_path(out_dir)}") + return out_dir + + +def _write_manifest( + out_dir: Path, source: str, version: str, repo: str, files: list[str] +) -> None: + """Record provenance next to the binaries (a cheap audit trail).""" + manifest = { + "source": source, + "version": version, + "repo": repo, + "url": f"https://github.com/{repo}/releases/tag/{version}", + "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "files": sorted(files), + } + (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") + + +def _link_local_assets(source: str, local_root: Path, bin_dir: Path) -> Path: + """Symlink/copy a local build tree into ``bin_dir/-local/``.""" + if source != "swarmit": + raise click.ClickException( + f"--fw-version local is only wired for source 'swarmit' so far, " + f"not '{source}'." + ) + local_root = local_root.expanduser().resolve() + out_dir = resolve_fw_root(bin_dir, source, "local") + out_dir.mkdir(parents=True, exist_ok=True) + mapping = { + "bootloader-dotbot-v3.hex": local_root + / "device/bootloader/Output/dotbot-v3/Debug/Exe/bootloader-dotbot-v3.hex", + "netcore-nrf5340-net.hex": local_root + / "device/network_core/Output/nrf5340-net/Debug/Exe/netcore-nrf5340-net.hex", + "03app_gateway_app-nrf5340-app.hex": local_root + / "mari/firmware/app/03app_gateway_app/Output/nrf5340-app/Debug/Exe/03app_gateway_app-nrf5340-app.hex", + "03app_gateway_net-nrf5340-net.hex": local_root + / "mari/firmware/app/03app_gateway_net/Output/nrf5340-net/Debug/Exe/03app_gateway_net-nrf5340-net.hex", + } + missing = [name for name, src in mapping.items() if not src.exists()] + if missing: + raise click.ClickException( + f"Missing local build artifacts: {', '.join(missing)}" + ) + for name, src in mapping.items(): + dest = out_dir / name + if dest.exists() or dest.is_symlink(): + dest.unlink() try: - size = download_file(url, out_dir / name) - except click.ClickException: - skipped += 1 - continue - click.echo(f" {name} ({_human_size(size)})") - if skipped: - click.echo(f" ({skipped} optional sample app(s) not in this release)") - count = len(assets) + len(example_bins) - skipped - click.echo(f"Done: {count} file(s) in {_short_path(out_dir)}") + os.symlink(src, dest) + click.echo(f"[LINK] {dest} -> {src}") + except OSError: + shutil.copy2(src, dest) + click.echo(f"[COPY] {dest} <- {src}") return out_dir @@ -524,14 +559,14 @@ def flash_role( calibration_hex = (bytes([count]) + matrices).hex() click.echo(f"[INFO] calibration: {count} matrices from {calibration_path}") - fw_root = resolve_fw_root(bin_dir, fw_version) + fw_root = resolve_fw_root(bin_dir, "swarmit", fw_version) # Auto-fetch: if the role's images aren't already present, pull the - # release into bin_dir// before flashing (npm-style). + # swarmit release into bin_dir/swarmit-/ before flashing. pre_app = fw_root / assets["app"] pre_net = fw_root / assets["net"] if fw_version != "local" and not (pre_app.exists() and pre_net.exists()): click.echo(f"[INFO] firmware {fw_version} not found in {fw_root}; fetching...") - fetch_assets(fw_version, bin_dir) + fetch_assets("swarmit", fw_version, bin_dir) if not fw_root.exists(): raise click.ClickException(f"Firmware root not found: {fw_root}") diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index 26ca75e6..ac173d4f 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -333,40 +333,45 @@ def test_intelhex_is_a_core_dependency(): assert flash.IntelHex is not None -def test_fetch_assets_skips_missing_optional_examples(tmp_path, monkeypatch): - """A 404 on an optional sample .bin must NOT abort the fetch — the four - required system images still complete (so provisioning's auto-fetch works - even when the sample apps aren't on the release).""" +def test_fetch_assets_downloads_release_into_source_version_dir(tmp_path, monkeypatch): + """fetch_assets pulls every .hex/.bin the release lists into + -/, skips .elf/.map, and writes a manifest.""" + import json as _json + import dotbot.firmware.flash as flash - downloaded = [] + fake_release = { + "tag_name": "0.8.0rc2", + "assets": [ + {"name": "bootloader-dotbot-v3.hex", "browser_download_url": "u1"}, + {"name": "netcore-nrf5340-net.hex", "browser_download_url": "u2"}, + {"name": "bootloader-dotbot-v3.elf", "browser_download_url": "u3"}, + ], + } + monkeypatch.setattr(flash, "resolve_release", lambda source, version: fake_release) def fake_download(url, dest): - name = url.rsplit("/", 1)[-1] - if name.endswith(".hex"): # the 4 required system images - dest.write_bytes(b"\x00") - downloaded.append(name) - return 1 # bytes written (download_file returns the size) - # optional sample .bin → simulate a release 404 - raise click.ClickException(f"HTTP Error 404: {name}") + dest.write_bytes(b"\x00") + return 1 monkeypatch.setattr(flash, "download_file", fake_download) - out = flash.fetch_assets("0.8.0rc1", tmp_path) # must not raise + out = flash.fetch_assets("swarmit", "latest", tmp_path) + assert out == tmp_path / "swarmit-0.8.0rc2" assert (out / "bootloader-dotbot-v3.hex").exists() assert (out / "netcore-nrf5340-net.hex").exists() - assert sum(n.endswith(".hex") for n in downloaded) == 4 + assert not (out / "bootloader-dotbot-v3.elf").exists() # .elf skipped + manifest = _json.loads((out / "manifest.json").read_text()) + assert manifest["source"] == "swarmit" + assert manifest["version"] == "0.8.0rc2" + assert "bootloader-dotbot-v3.hex" in manifest["files"] -def test_fetch_assets_still_fails_on_missing_system_image(tmp_path, monkeypatch): - """A 404 on a REQUIRED system .hex stays fatal (bad version tag).""" +def test_fetch_assets_unknown_source_errors(tmp_path): + """An unknown source is a clear error, not a KeyError.""" import dotbot.firmware.flash as flash - def fake_download(url, dest): - raise click.ClickException("HTTP Error 404") - - monkeypatch.setattr(flash, "download_file", fake_download) with pytest.raises(click.ClickException): - flash.fetch_assets("0.0.0-nope", tmp_path) + flash.fetch_assets("not-a-source", "latest", tmp_path) def test_resolve_latest_version_returns_newest_tag(monkeypatch): From 670038d7d38ad23f20d897a4ad402128d7a210b6 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 13:08:18 +0200 Subject: [PATCH 11/27] dotbot/firmware: download fetch assets in parallel AI-assisted: Claude Opus 4.8 --- dotbot/firmware/flash.py | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 402d471f..cf533a05 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -17,6 +17,7 @@ import time import urllib.error import urllib.request +from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime, timezone from pathlib import Path @@ -428,12 +429,33 @@ def fetch_assets( raise click.ClickException( f"{source} release {tag} publishes no .hex/.bin assets." ) - click.echo(f"Fetching {source} {tag} into {_short_path(out_dir)}") - names = [] - for asset in assets: - size = download_file(asset["browser_download_url"], out_dir / asset["name"]) - click.echo(f" {asset['name']} ({_human_size(size)})") - names.append(asset["name"]) + click.echo( + f"Fetching {source} {tag} ({len(assets)} files) into {_short_path(out_dir)}" + ) + names: list[str] = [] + errors: list[str] = [] + # Downloads are I/O-bound, so a small thread pool overlaps them (urllib + # releases the GIL during the network read). Sources stay sequential. + with ThreadPoolExecutor(max_workers=8) as pool: + future_to_name = { + pool.submit( + download_file, asset["browser_download_url"], out_dir / asset["name"] + ): asset["name"] + for asset in assets + } + for future in as_completed(future_to_name): + name = future_to_name[future] + try: + size = future.result() + except click.ClickException as exc: + errors.append(f"{name}: {exc.format_message()}") + continue + names.append(name) + click.echo(f" {name} ({_human_size(size)})") + if errors: + raise click.ClickException( + f"{len(errors)} asset(s) failed to download:\n " + "\n ".join(errors) + ) _write_manifest(out_dir, source, tag, RELEASE_SOURCES[source], names) click.echo(f"Done: {len(names)} file(s) in {_short_path(out_dir)}") return out_dir From 1433a37c8841db344a37b83a7c6a10e669a7b88d Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 14:27:20 +0200 Subject: [PATCH 12/27] dotbot/cli: announce the active config file at startup AI-assisted: Claude Opus 4.8 --- dotbot/cli/main.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/dotbot/cli/main.py b/dotbot/cli/main.py index e7a3ba81..9d3a228b 100644 --- a/dotbot/cli/main.py +++ b/dotbot/cli/main.py @@ -128,6 +128,11 @@ def cli(ctx, config_path, deployment_name): except ConfigError as exc: raise click.ClickException(str(exc)) from exc + if path is not None: + click.echo(f"using config file at {path}", err=True) + else: + click.echo("no config file found; using built-in defaults", err=True) + ctx.obj["config"] = config ctx.obj["config_path"] = path ctx.obj["deployment"] = deployment From 2a51c20a755087dd397b49d9451fe37be0cb9489 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 14:38:26 +0200 Subject: [PATCH 13/27] dotbot/firmware: retry transient download failures AI-assisted: Claude Opus 4.8 --- dotbot/firmware/flash.py | 63 ++++++++++++++++++++++++++----------- dotbot/tests/test_device.py | 41 ++++++++++++++++++++++++ 2 files changed, 85 insertions(+), 19 deletions(-) diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index cf533a05..51f28d38 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -144,25 +144,48 @@ def _short_path(path: Path) -> str: return rel if not rel.startswith("..") else str(path) -def download_file(url: str, dest: Path) -> int: - """Download ``url`` to ``dest``; return the number of bytes written.""" - try: - with urllib.request.urlopen(url) as resp: - status = getattr(resp, "status", 200) - if status != 200: - raise click.ClickException(f"HTTP {status} while downloading {url}") - data = resp.read() - except urllib.error.HTTPError as exc: - raise click.ClickException( - f"HTTP error while downloading {url}: {exc}" - ) from exc - except urllib.error.URLError as exc: - raise click.ClickException( - f"Network error while downloading {url}: {exc}" - ) from exc +# Transient HTTP statuses worth retrying (GitHub's asset CDN 502s now and +# then under concurrent load; 429 is rate-limiting). +_RETRY_STATUS = {429, 500, 502, 503, 504} + + +def download_file(url: str, dest: Path, *, retries: int = 3) -> int: + """Download ``url`` to ``dest``; return the number of bytes written. - dest.write_bytes(data) - return len(data) + Retries transient failures (connection errors and HTTP 429/5xx) with + exponential backoff - GitHub's CDN occasionally 502s under concurrent + downloads, and a sporadic failure shouldn't abort the whole fetch. + """ + for attempt in range(retries + 1): + try: + with urllib.request.urlopen(url) as resp: + status = getattr(resp, "status", 200) + if status != 200: + raise click.ClickException(f"HTTP {status} while downloading {url}") + data = resp.read() + dest.write_bytes(data) + return len(data) + except urllib.error.HTTPError as exc: + if exc.code not in _RETRY_STATUS or attempt == retries: + raise click.ClickException( + f"HTTP error while downloading {url}: {exc}" + ) from exc + reason = f"HTTP {exc.code}" + except urllib.error.URLError as exc: + if attempt == retries: + raise click.ClickException( + f"Network error while downloading {url}: {exc}" + ) from exc + reason = str(exc.reason) + delay = 0.5 * (2**attempt) + click.echo( + f" [retry] {url.rsplit('/', 1)[-1]} ({reason}); retrying in {delay:.1f}s", + err=True, + ) + time.sleep(delay) + raise click.ClickException( # pragma: no cover - loop always returns/raises + f"Failed to download {url} after {retries} retries." + ) def _github_get(url: str): @@ -436,7 +459,9 @@ def fetch_assets( errors: list[str] = [] # Downloads are I/O-bound, so a small thread pool overlaps them (urllib # releases the GIL during the network read). Sources stay sequential. - with ThreadPoolExecutor(max_workers=8) as pool: + # Kept modest: GitHub's asset CDN starts 502ing under heavier fan-out + # (download_file retries transient failures regardless). + with ThreadPoolExecutor(max_workers=4) as pool: future_to_name = { pool.submit( download_file, asset["browser_download_url"], out_dir / asset["name"] diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index ac173d4f..3d9941f0 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -409,3 +409,44 @@ def boom(req): monkeypatch.setattr(flash.urllib.request, "urlopen", boom) with pytest.raises(click.ClickException): flash.resolve_latest_version() + + +def test_download_file_retries_transient_5xx(tmp_path, monkeypatch): + """A sporadic 502 (GitHub's CDN under concurrent load) is retried, then + succeeds - one bad gateway shouldn't abort the whole fetch.""" + import io + + import dotbot.firmware.flash as flash + + calls = {"n": 0} + + def flaky_urlopen(url): + calls["n"] += 1 + if calls["n"] == 1: + raise flash.urllib.error.HTTPError(url, 502, "Bad Gateway", {}, None) + return io.BytesIO(b"\xde\xad") + + monkeypatch.setattr(flash.urllib.request, "urlopen", flaky_urlopen) + monkeypatch.setattr(flash.time, "sleep", lambda _delay: None) # skip real backoff + + dest = tmp_path / "spin-dotbot-v3.hex" + size = flash.download_file("http://x/spin-dotbot-v3.hex", dest, retries=3) + assert size == 2 + assert dest.read_bytes() == b"\xde\xad" + assert calls["n"] == 2 # one retry + + +def test_download_file_gives_up_on_non_transient(tmp_path, monkeypatch): + """A 404 is not transient - it surfaces immediately, with no backoff.""" + import dotbot.firmware.flash as flash + + def not_found(url): + raise flash.urllib.error.HTTPError(url, 404, "Not Found", {}, None) + + sleeps: list[float] = [] + monkeypatch.setattr(flash.urllib.request, "urlopen", not_found) + monkeypatch.setattr(flash.time, "sleep", lambda d: sleeps.append(d)) + + with pytest.raises(click.ClickException): + flash.download_file("http://x/missing.hex", tmp_path / "missing.hex", retries=3) + assert sleeps == [] # never retried From 1f170fce743c47c9b0024ffd62524828dde378ea Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 14:55:18 +0200 Subject: [PATCH 14/27] dotbot/cli: summarize fetched firmware folders at the end AI-assisted: Claude Opus 4.8 --- dotbot/cli/fw.py | 12 ++++++++++-- dotbot/firmware/flash.py | 1 - 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index 76a9fc7e..24480040 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -296,7 +296,11 @@ def fetch(source, fw_version, local_root): swarm system images) and DotBot-firmware (bare + sandbox apps). The two version independently, so pinning a version with -f requires a --source. """ - from dotbot.firmware.flash import DEFAULT_FETCH_SOURCES, fetch_assets + from dotbot.firmware.flash import ( + DEFAULT_FETCH_SOURCES, + _short_path, + fetch_assets, + ) if fw_version is not None and source is None: raise click.ClickException( @@ -304,11 +308,15 @@ def fetch(source, fw_version, local_root): "version independently." ) sources = [source] if source else list(DEFAULT_FETCH_SOURCES) + fetched: list[Path] = [] for src in sources: version = fw_version or "latest" if version == "latest": click.echo(f"Fetching the latest {src} release...") - fetch_assets(src, version, artifacts_dir(), local_root) + fetched.append(fetch_assets(src, version, artifacts_dir(), local_root)) + click.echo("\nDone. Firmware fetched into:") + for path in fetched: + click.echo(f" {_short_path(path)}") @cmd.command(name="list") diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 51f28d38..5a2122f2 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -482,7 +482,6 @@ def fetch_assets( f"{len(errors)} asset(s) failed to download:\n " + "\n ".join(errors) ) _write_manifest(out_dir, source, tag, RELEASE_SOURCES[source], names) - click.echo(f"Done: {len(names)} file(s) in {_short_path(out_dir)}") return out_dir From 32ea774b9aa5758c7228b909eb56a1be07be1ca7 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:27:05 +0200 Subject: [PATCH 15/27] dotbot/cli: centralize the default artifacts-cache path AI-assisted: Claude Opus 4.8 --- dotbot/cli/_artifacts.py | 28 +++++++++++++++++++--------- dotbot/cli/device.py | 9 +++++---- dotbot/cli/main.py | 2 +- 3 files changed, 25 insertions(+), 14 deletions(-) diff --git a/dotbot/cli/_artifacts.py b/dotbot/cli/_artifacts.py index 9b87c681..b5516a78 100644 --- a/dotbot/cli/_artifacts.py +++ b/dotbot/cli/_artifacts.py @@ -3,9 +3,9 @@ """Shared artifact-resolution + friendly-error helpers for `fw` / `device`. -Owns the CWD-local ``./artifacts/`` convention, the absolute-path echo on -every cache read/write (so running from a random directory never silently -touches a relative path the user didn't name), the auto-resolve decision +Owns the user-level ``~/.dotbot/artifacts/`` cache convention, the +absolute-path echo on every cache read/write (so you always see where files +landed, regardless of the directory you ran from), the auto-resolve decision tree used by ``dotbot device flash `` (present → build → error), and the two centralized tool-missing messages (SES for builds, nrfjprog for device ops). @@ -20,6 +20,14 @@ import click +# Single source of truth for the default cache location, so changing it is a +# one-line edit. `artifacts_dir()` is the behavioral path (env-overridable, ~ +# expanded, resolved); `DEFAULT_ARTIFACTS_DISPLAY` is the human-readable form +# for help text. (Docstrings and the Markdown docs can't interpolate a +# variable, so they spell the path out literally - keep them in sync by hand.) +_DEFAULT_ARTIFACTS_PARTS = (".dotbot", "artifacts") +DEFAULT_ARTIFACTS_DISPLAY = "~/" + "/".join(_DEFAULT_ARTIFACTS_PARTS) + def artifacts_dir() -> Path: """The firmware cache: ``~/.dotbot/artifacts/`` by default. @@ -29,7 +37,9 @@ def artifacts_dir() -> Path: the launch dir stays clean. Override with ``$DOTBOT_ARTIFACTS_DIR``. """ override = os.environ.get("DOTBOT_ARTIFACTS_DIR") - base = Path(override) if override else Path.home() / ".dotbot" / "artifacts" + base = ( + Path(override) if override else Path.home().joinpath(*_DEFAULT_ARTIFACTS_PARTS) + ) return base.expanduser().resolve() @@ -101,13 +111,13 @@ def resolve_app_artifact( ) -> Path: """Auto-resolve a single app's firmware artifact for cable-flashing. - Decision tree (npm-style): present in ``./artifacts/`` → build from - source → clear error pointing at build/fetch. An *explicit file path* + Decision tree (npm-style): present in ``~/.dotbot/artifacts/`` → build + from source → clear error pointing at build/fetch. An *explicit file path* is handled by the caller before this is reached. - - Flat ``./artifacts/-.hex`` (bare) or - ``./artifacts/-sandbox-.bin`` (sandbox), as produced by - `dotbot fw artifacts`. + - Flat ``-.hex`` (bare) or ``-sandbox-.bin`` + (sandbox), as produced by `dotbot fw artifacts`, found across the + cache's ``-/`` and ``-local/`` dirs. - Else, if a DotBot-firmware repo is locatable, build it (needs SES) and use the SES output path. - Else, a friendly error telling the user to build or fetch first. diff --git a/dotbot/cli/device.py b/dotbot/cli/device.py index 774645b3..22d7603e 100644 --- a/dotbot/cli/device.py +++ b/dotbot/cli/device.py @@ -19,6 +19,7 @@ import click from dotbot.cli._artifacts import ( + DEFAULT_ARTIFACTS_DISPLAY, artifacts_dir, ensure_nrfjprog, resolve_app_artifact, @@ -54,7 +55,7 @@ def _looks_like_path(value: str) -> bool: show_default=True, help=( "Target board: selects the chip family + core to flash (nRF52 vs " - "nRF5340 app/net) and resolves - in ./artifacts/." + f"nRF5340 app/net) and resolves - in {DEFAULT_ARTIFACTS_DISPLAY}/." ), ) @click.option("--sandbox", is_flag=True, help="Resolve the sandbox-app flavor (.bin).") @@ -70,7 +71,7 @@ def _looks_like_path(value: str) -> bool: def flash(ctx, app, sn_starting_digits, board, sandbox, config): """Flash a firmware image to one cabled device (whole-chip program). - APP is an app name (resolved against ./artifacts/, building from source + APP is an app name (resolved against ~/.dotbot/artifacts/, building from source if needed) or an explicit `.hex`/`.bin` file path. `--board` selects the chip family + core to program (see `dotbot fw targets`); no sandbox host is required. @@ -99,7 +100,7 @@ def _fw_version_option(f): default=None, help=( "Release version to flash, e.g. 0.8.0rc2 (default: latest swarmit " - "release). Binaries are fetched into ./artifacts/ if not cached." + f"release). Binaries are fetched into {DEFAULT_ARTIFACTS_DISPLAY}/ if not cached." ), )(f) @@ -133,7 +134,7 @@ def flash_swarmit_sandbox( Flashes the SwarmIT bootloader (app core) + netcore + writes the network identity. Auto-fetches the release if not already in - ./artifacts//. + ~/.dotbot/artifacts/swarmit-/. """ from dotbot.firmware.flash import ( flash_role, diff --git a/dotbot/cli/main.py b/dotbot/cli/main.py index 9d3a228b..8fca3951 100644 --- a/dotbot/cli/main.py +++ b/dotbot/cli/main.py @@ -5,7 +5,7 @@ The top level is the four object-namespaces, each one *kind of thing*: - fw — firmware artifacts (files in ./artifacts/, no hardware) + fw — firmware artifacts (cached in ~/.dotbot/artifacts/, no hardware) device — one connected device (cable / probe) swarm — the fleet (radio / OTA) run — host-side processes (software you launch on your computer) From 82112633808f3a7fb6e879b6c6823f681bdda736 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:27:06 +0200 Subject: [PATCH 16/27] dotbot/firmware: pin fetched firmware versions to pydotbot `dotbot fw fetch` with no args now resolves the pinned versions (swarmit from the installed package, dotbot-firmware from a declared constant) instead of the latest release; pass `-f latest` for the old behavior. AI-assisted: Claude Opus 4.8 --- dotbot/cli/fw.py | 65 +++++++++++++++++++++------------ dotbot/firmware/flash.py | 36 +++++++++++++++++++ dotbot/tests/test_device.py | 72 +++++++++++++++++++++++++++++++++++++ 3 files changed, 150 insertions(+), 23 deletions(-) diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index 24480040..6e47395f 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -9,22 +9,24 @@ - `build` compiles from source via SES (`emBuild`) in `DotBot-firmware`, leaving the result in the SES `Output/.../Exe/` tree and echoing that - path — it does *not* copy into `./artifacts/`. Bare apps by default; + path — it does *not* copy into the cache. Bare apps by default; `--sandbox` builds the TrustZone NS flavor (`sandbox-`, `.bin`). -- `artifacts` builds *and* collects the result into `./artifacts/`, with - the flat `-.hex` / `-sandbox-.bin` names. -- `fetch` downloads a published release into `./artifacts//`. -- `list` shows what's cached in `./artifacts/`. +- `artifacts` builds *and* collects the result into the cache + (`~/.dotbot/artifacts/dotbot-firmware-local/`), with the flat + `-.hex` / `-sandbox-.bin` names. +- `fetch` downloads the pinned release (or a `-f `/`latest` + override) into `~/.dotbot/artifacts/-/`. +- `list` shows what's cached in `~/.dotbot/artifacts/`. - `make` is the low-level escape hatch: it forwards arbitrary arguments to `make` in the firmware repo (workspace-resolved SEGGER_DIR) for the Makefile knobs `build` deliberately doesn't model. -Only `artifacts` and `fetch` populate `./artifacts/`. The device-flash +Only `artifacts` and `fetch` populate the cache. The device-flash commands then auto-resolve their input, by *different* rules: `dotbot -device flash ` resolves an app image present-in-`./artifacts/` → +device flash ` resolves an app image present in `~/.dotbot/artifacts/` → build-from-source → error (it never fetches); `device flash-swarmit-sandbox` -/ `flash-mari-gateway` resolve a release's system firmware -present-in-`./artifacts/` → fetch (they never build). +/ `flash-mari-gateway` resolve a release's system firmware present in +`~/.dotbot/artifacts/` → fetch (they never build). """ import sys @@ -32,7 +34,11 @@ import click -from dotbot.cli._artifacts import artifacts_dir, echo_artifact_path +from dotbot.cli._artifacts import ( + DEFAULT_ARTIFACTS_DISPLAY, + artifacts_dir, + echo_artifact_path, +) from dotbot.cli._cfg import from_config from dotbot.cli._fw_helpers import ( BARE_TARGETS, @@ -211,7 +217,7 @@ def list_targets(sandbox): "out_dir", type=click.Path(file_okay=False, dir_okay=True), default=None, - help="Where to collect artifacts. Default: ./artifacts/ (your CWD).", + help=f"Where to collect artifacts. Default: {DEFAULT_ARTIFACTS_DISPLAY}/dotbot-firmware-local/.", ) @click.option( "--print-path", @@ -276,13 +282,13 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos "-S", type=click.Choice(list(("swarmit", "dotbot-firmware"))), default=None, - help="Limit to one source (default: fetch the latest from all sources).", + help="Limit to one source (default: fetch the pinned version from all sources).", ) @click.option( "--fw-version", "-f", default=None, - help="Release tag for --source (default: latest), or 'local'.", + help="Override the pinned version for --source: a release tag, 'latest', or 'local'.", ) @click.option( "--local-root", @@ -290,16 +296,21 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos help="Root of a local build tree (with --source --fw-version local).", ) def fetch(source, fw_version, local_root): - """Download released firmware into ~/.dotbot/artifacts/-/. - - With no flags, fetches the latest release from every source: swarmit (the - swarm system images) and DotBot-firmware (bare + sandbox apps). The two - version independently, so pinning a version with -f requires a --source. + """Download firmware into ~/.dotbot/artifacts/-/. + + With no flags, fetches the exact release this pydotbot is pinned to, from + every source: swarmit (swarm system images, version inferred from the + installed swarmit package) and DotBot-firmware (bare + sandbox apps, the + version pydotbot is tested against). The two version independently, so + overriding with -f requires a --source - pass `-f latest` for the newest + release or `-f ` for a specific one. """ + from dotbot import pydotbot_version from dotbot.firmware.flash import ( DEFAULT_FETCH_SOURCES, _short_path, fetch_assets, + pinned_version, ) if fw_version is not None and source is None: @@ -310,9 +321,15 @@ def fetch(source, fw_version, local_root): sources = [source] if source else list(DEFAULT_FETCH_SOURCES) fetched: list[Path] = [] for src in sources: - version = fw_version or "latest" - if version == "latest": - click.echo(f"Fetching the latest {src} release...") + if fw_version is None: + version = pinned_version(src) + click.echo( + f"Fetching {src} {version} (pinned by pydotbot {pydotbot_version()})..." + ) + else: + version = fw_version + if version == "latest": + click.echo(f"Fetching the latest {src} release...") fetched.append(fetch_assets(src, version, artifacts_dir(), local_root)) click.echo("\nDone. Firmware fetched into:") for path in fetched: @@ -321,11 +338,13 @@ def fetch(source, fw_version, local_root): @cmd.command(name="list") def list_artifacts(): - """List firmware artifacts cached in ./artifacts/.""" + """List firmware artifacts cached in ~/.dotbot/artifacts/.""" root = artifacts_dir() echo_artifact_path(root, action="listing") if not root.is_dir(): - click.echo("(no ./artifacts/ yet — run `dotbot fw build` or `dotbot fw fetch`)") + click.echo( + "(nothing cached yet — run `dotbot fw fetch` or `dotbot fw artifacts`)" + ) return found = sorted( p for p in root.rglob("*") if p.is_file() and p.suffix in (".hex", ".bin") diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 5a2122f2..0004a478 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -66,6 +66,13 @@ "dotbot-firmware": "DotBots/DotBot-firmware", } DEFAULT_FETCH_SOURCES = ("swarmit", "dotbot-firmware") +# The DotBot-firmware release this pydotbot is built and tested against. Unlike +# swarmit (a Python dependency, whose firmware release is tagged identically to +# the package, so we read it from importlib.metadata), DotBot-firmware is not a +# Python package - so the expected version is declared here and bumped +# deliberately when pydotbot adopts a new release. `dotbot fw fetch` (no -f) +# pulls exactly this; -f overrides it. +DOTBOT_FIRMWARE_VERSION = "1.22.0rc1" # Application images are linked after the bootloader. APP_FLASH_BASE_ADDR = 0x00010000 # Programmer bring-up files @@ -228,6 +235,32 @@ def resolve_latest_version(source: str = "swarmit") -> str: return resolve_release(source, "latest")["tag_name"] +def pinned_version(source: str) -> str: + """The exact firmware release this pydotbot pins for ``source``. + + swarmit is a Python dependency whose firmware release is tagged identically + to the package, so its pin is read from the installed package. DotBot-firmware + is not a Python package, so its pin is the declared ``DOTBOT_FIRMWARE_VERSION``. + `dotbot fw fetch` (no -f) resolves to these; pass -f to override. + """ + if source == "swarmit": + from importlib.metadata import PackageNotFoundError + from importlib.metadata import version as _pkg_version + + try: + return _pkg_version("swarmit") + except PackageNotFoundError as exc: + raise click.ClickException( + "Cannot infer the swarmit firmware version: the swarmit package " + "is not installed. Pass -f explicitly." + ) from exc + if source == "dotbot-firmware": + return DOTBOT_FIRMWARE_VERSION + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + + def convert_bin_to_hex(bin_path: Path, base_addr: int) -> Path: if IntelHex is None: raise click.ClickException( @@ -489,11 +522,14 @@ def _write_manifest( out_dir: Path, source: str, version: str, repo: str, files: list[str] ) -> None: """Record provenance next to the binaries (a cheap audit trail).""" + from dotbot import pydotbot_version + manifest = { "source": source, "version": version, "repo": repo, "url": f"https://github.com/{repo}/releases/tag/{version}", + "pydotbot": pydotbot_version(), "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), "files": sorted(files), } diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index 3d9941f0..d0c76f9d 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -10,6 +10,8 @@ board), and the friendly nrfjprog-missing error. """ +from pathlib import Path + import click import pytest from click.testing import CliRunner @@ -364,6 +366,7 @@ def fake_download(url, dest): assert manifest["source"] == "swarmit" assert manifest["version"] == "0.8.0rc2" assert "bootloader-dotbot-v3.hex" in manifest["files"] + assert manifest["pydotbot"] # provenance: which pydotbot fetched this def test_fetch_assets_unknown_source_errors(tmp_path): @@ -450,3 +453,72 @@ def not_found(url): with pytest.raises(click.ClickException): flash.download_file("http://x/missing.hex", tmp_path / "missing.hex", retries=3) assert sleeps == [] # never retried + + +def test_pinned_version_dotbot_firmware_is_declared(): + """DotBot-firmware (not a Python dep) pins to the declared constant.""" + import dotbot.firmware.flash as flash + + assert flash.pinned_version("dotbot-firmware") == flash.DOTBOT_FIRMWARE_VERSION + + +def test_pinned_version_swarmit_from_installed_package(): + """swarmit's firmware version is inferred from the installed package.""" + import importlib.metadata as md + + import dotbot.firmware.flash as flash + + assert flash.pinned_version("swarmit") == md.version("swarmit") + + +def test_pinned_version_unknown_source_errors(): + """An unknown source is a clear error, not a KeyError.""" + import dotbot.firmware.flash as flash + + with pytest.raises(click.ClickException): + flash.pinned_version("not-a-source") + + +def test_fetch_no_args_resolves_pinned_versions(monkeypatch): + """`dotbot fw fetch` with no flags fetches the pinned version per source, + not 'latest'.""" + import dotbot.firmware.flash as flash + from dotbot.cli.fw import cmd as fw_cmd + + calls: list[tuple[str, str]] = [] + monkeypatch.setattr(flash, "pinned_version", lambda src: f"PIN-{src}") + monkeypatch.setattr( + flash, + "fetch_assets", + lambda src, version, bin_dir, local_root=None: ( + calls.append((src, version)) or Path(f"/x/{src}-{version}") + ), + ) + res = CliRunner().invoke(fw_cmd, ["fetch"]) + assert res.exit_code == 0, res.output + assert calls == [ + ("swarmit", "PIN-swarmit"), + ("dotbot-firmware", "PIN-dotbot-firmware"), + ] + assert "latest" not in res.output # the pinned path never says "latest" + + +def test_fetch_explicit_version_overrides_pin(monkeypatch): + """-f with --source bypasses the pin and passes through verbatim.""" + import dotbot.firmware.flash as flash + from dotbot.cli.fw import cmd as fw_cmd + + calls: list[tuple[str, str]] = [] + pin_called: list[str] = [] + monkeypatch.setattr(flash, "pinned_version", lambda src: pin_called.append(src)) + monkeypatch.setattr( + flash, + "fetch_assets", + lambda src, version, bin_dir, local_root=None: ( + calls.append((src, version)) or Path(f"/x/{src}-{version}") + ), + ) + res = CliRunner().invoke(fw_cmd, ["fetch", "-S", "dotbot-firmware", "-f", "1.21.0"]) + assert res.exit_code == 0, res.output + assert calls == [("dotbot-firmware", "1.21.0")] + assert pin_called == [] # explicit -f never consults the pin From 7a2abea45577bc48ac4310fb5fe07bb7f777f8b3 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:27:06 +0200 Subject: [PATCH 17/27] doc: refresh firmware-cache location and pinned fetch AI-assisted: Claude Opus 4.8 --- doc/cli/device.md | 7 +++-- doc/cli/fw.md | 50 ++++++++++++++++++++++++++-------- doc/cli/index.md | 4 +-- doc/cli/swarm.md | 6 ++-- doc/guides/one-bot.md | 2 +- doc/reference/configuration.md | 2 +- 6 files changed, 50 insertions(+), 21 deletions(-) diff --git a/doc/cli/device.md b/doc/cli/device.md index 5a06bef1..4cf5fc9a 100644 --- a/doc/cli/device.md +++ b/doc/cli/device.md @@ -32,8 +32,9 @@ UART↔MQTT bridge process is a different thing - that's [`run gateway`](run.md) ## Flash an app -`flash` resolves `` to `./artifacts/-.hex`, **building it if the -file isn't there**; an explicit `.hex`/`.bin` path is flashed as-is. +`flash` resolves `` to a cached `-.hex` (under +`~/.dotbot/artifacts/`), **building it if the file isn't there**; an explicit +`.hex`/`.bin` path is flashed as-is. ```bash export DOTBOT_FIRMWARE_REPO=$(pwd)/repos/DotBot-firmware @@ -86,7 +87,7 @@ dotbot device flash nrf5340_net -b nrf5340dk-net -s 10 `flash-mari-gateway` and `flash-swarmit-sandbox` flash a **complete system firmware** (both cores) and write the **network identity** in one shot. They auto-fetch the -named release into `./artifacts/` if it isn't cached. +named release into `~/.dotbot/artifacts/` if it isn't cached. ```bash # nRF5340-DK → swarm gateway diff --git a/doc/cli/fw.md b/doc/cli/fw.md index 5be7f63e..e3eeca2c 100644 --- a/doc/cli/fw.md +++ b/doc/cli/fw.md @@ -43,8 +43,8 @@ segger_dir = "/path/to/SEGGER Embedded Studio X.YY" | Goal | Command | |---|---| | Compile an app, leave it in the SES `Output/` tree (path echoed) | `dotbot fw build` | -| Compile **and** collect a flat `-.hex` into `./artifacts/` | `dotbot fw artifacts` | -| Download a published release (latest by default) into `./artifacts//` | `dotbot fw fetch [-f ]` | +| Compile **and** collect a flat `-.hex` into the cache | `dotbot fw artifacts` | +| Download the pinned release(s) into `~/.dotbot/artifacts/-/` | `dotbot fw fetch [-S -f ]` | | List targets you can build | `dotbot fw targets [--sandbox]` | | List what's cached locally | `dotbot fw list` | | A Makefile knob the CLI doesn't model | `dotbot fw make ` | @@ -52,9 +52,10 @@ segger_dir = "/path/to/SEGGER Embedded Studio X.YY" **`build` vs `artifacts`**: both compile via SES. `build` stops once SES is done (output stays buried in the per-target `Output/` tree). `artifacts` goes one step further and copies a flat, predictably-named `-.hex` into -`./artifacts/` - which is exactly where `dotbot device flash ` and the -swarm tools look. Reach for `artifacts` when you intend to flash; `build` when -you only want to know it compiles. +the cache (`~/.dotbot/artifacts/dotbot-firmware-local/`) - which is exactly +where `dotbot device flash ` and the swarm tools look. Reach for +`artifacts` when you intend to flash; `build` when you only want to know it +compiles. ## `build` / `artifacts` flags @@ -69,8 +70,8 @@ Both share the same build options: | `--rebuild` | Force a full rebuild (default: incremental) | | `-v, --verbose` | Full SES output | -`artifacts` adds `--out ` (default `./artifacts/`) and `--print-path` -(report where the artifact would land without building). See +`artifacts` adds `--out ` (default `~/.dotbot/artifacts/dotbot-firmware-local/`) +and `--print-path` (report where the artifact would land without building). See `dotbot fw --help` for the full list. > **Flag mismatch to remember:** `fw` selects a board with `--target/-t`, but @@ -105,12 +106,40 @@ Notes: - The nRF5340 radio lives on the **net core**, so a gateway needs two images: `dotbot_gateway` on `nrf5340dk-app` **and** `nrf5340_net` on `nrf5340dk-net`. +## `fetch` - pinned firmware + +`dotbot fw fetch` downloads prebuilt firmware from two release sources - +**swarmit** (the swarm system images) and **DotBot-firmware** (the bare and +sandbox apps) - into `~/.dotbot/artifacts/-/`, each with a +`manifest.json` recording where it came from. + +With no flags it fetches the **exact versions this `dotbot` is pinned to**, so a +given `dotbot` always pulls a known-good, reproducible set: + +- **swarmit** is also a Python dependency, so its firmware version is read from + the installed `swarmit` package. +- **DotBot-firmware** is not a Python package, so the version `dotbot` is built + and tested against is declared in `dotbot` and bumped deliberately. + +Override per source when you need something else - the two version +independently, so `-f` requires `-S`: + +```bash +dotbot fw fetch # pinned versions, both sources +dotbot fw fetch -S dotbot-firmware -f latest # newest dotbot-firmware release +dotbot fw fetch -S swarmit -f 0.8.0rc2 # a specific swarmit version +``` + +The cache is user-level and shared across projects (override the location with +`$DOTBOT_ARTIFACTS_DIR`); `dotbot device flash` and the swarm tools resolve +their input from it. + ## Examples ```bash export DOTBOT_FIRMWARE_REPO=/path/to/DotBot-firmware -# Bare DotBot app for a DotBot v3 → ./artifacts/dotbot-dotbot-v3.hex +# Bare DotBot app for a DotBot v3 → ~/.dotbot/artifacts/dotbot-firmware-local/dotbot-dotbot-v3.hex dotbot fw artifacts --app dotbot # Just confirm an app compiles (no collection) @@ -123,9 +152,8 @@ dotbot fw artifacts --app nrf5340_net -t nrf5340dk-net # Sandbox (NS) "spin" app for a DotBot v3 → .bin, for OTA via swarm dotbot fw artifacts --app spin -t dotbot-v3 --sandbox -# Pull a published release instead of building → ./artifacts// -dotbot fw fetch # latest swarmit release (prereleases included) -dotbot fw fetch -f 0.8.0rc2 # or pin an explicit version +# Download released firmware instead of building (see "fetch" above) +dotbot fw fetch # the versions this dotbot is pinned to (both sources) # See what's cached dotbot fw list diff --git a/doc/cli/index.md b/doc/cli/index.md index 0dc389f6..c763f1ac 100644 --- a/doc/cli/index.md +++ b/doc/cli/index.md @@ -46,8 +46,8 @@ Need a process running on my computer (UI, gateway bridge, demo)? ─► run A few signposts so the namespaces don't blur together: - **`fw` never touches hardware.** It only produces or lists artifacts in - `./artifacts/`. Flashing always happens under `device` (cabled) or `swarm` - (OTA). + `~/.dotbot/artifacts/`. Flashing always happens under `device` (cabled) or + `swarm` (OTA). - **Bare vs. sandbox artifacts.** `fw` builds bare apps (`.hex`) by default; `fw artifacts --sandbox` builds TrustZone apps (`.bin`) - the payload `swarm` flashes over the air. diff --git a/doc/cli/swarm.md b/doc/cli/swarm.md index f9e7cdfb..17d47ccd 100644 --- a/doc/cli/swarm.md +++ b/doc/cli/swarm.md @@ -46,8 +46,8 @@ The OTA payload is a **sandbox** app - a TrustZone non-secure `.bin`. Build it, or fetch a pre-compiled release: ```bash -dotbot fw artifacts --sandbox # builds -> ./artifacts/-sandbox-.bin -dotbot fw fetch -f 0.8.0rc1 # or pull from a release into ./artifacts// +dotbot fw artifacts --sandbox # builds -> ~/.dotbot/artifacts/dotbot-firmware-local/-sandbox-.bin +dotbot fw fetch # or pull the pinned releases into ~/.dotbot/artifacts/-/ ``` Sandbox apps include `dotbot`, `move`, `rgbled`, `spin`, `timer`. Artifact @@ -83,7 +83,7 @@ to override). If the broker needs auth, set `DOTBOT_MQTT_USER` / ```bash dotbot swarm status # who's out there + their state dotbot swarm status -w # keep watching -dotbot swarm flash ./artifacts/spin-sandbox-dotbot-v3.bin -ys +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-*/spin-sandbox-dotbot-v3.bin -ys dotbot swarm stop # back to bootloader (before re-flashing) dotbot swarm start # (re)start the loaded app dotbot swarm monitor # tail SWARMIT_EVENT_LOG from bots diff --git a/doc/guides/one-bot.md b/doc/guides/one-bot.md index 6fe78cf3..5d56c066 100644 --- a/doc/guides/one-bot.md +++ b/doc/guides/one-bot.md @@ -14,7 +14,7 @@ The DotBot v3 is an nRF5340, which has two cores - the application core (your app) and the network core (the radio) - so you build and flash two images: ```bash -# build the bare dotbot apps into ./artifacts/ (needs SEGGER Embedded Studio) +# build the bare dotbot apps into the cache (needs SEGGER Embedded Studio) dotbot fw artifacts --app dotbot dotbot fw artifacts --app nrf5340_net --target nrf5340dk-net # cable-flash to the bot whose J-Link serial starts with 77 diff --git a/doc/reference/configuration.md b/doc/reference/configuration.md index f57c8f44..6c5035e4 100644 --- a/doc/reference/configuration.md +++ b/doc/reference/configuration.md @@ -220,7 +220,7 @@ default_deployment = "inria" # used when --deployment / DOTBOT_D conn = "mqtts://broker.local:8883" swarm_id = "0001" log_level = "info" -artifacts_dir = "./artifacts" +artifacts_dir = "~/.dotbot/artifacts" # A physical deployment. Select it with `--deployment inria`, DOTBOT_DEPLOYMENT, or # default_deployment above - don't edit this table to switch deployments. From 0e138eba8d4f2039f2687ecd822f03dbbadf1865 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:27:06 +0200 Subject: [PATCH 18/27] readme: point fetch + swarm-flash examples at the firmware cache AI-assisted: Claude Opus 4.8 --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 2d8bf98c..240b8b83 100644 --- a/README.md +++ b/README.md @@ -95,13 +95,13 @@ We also need a more powerful gateway firmware. Let's flash both - the network id comes from your config: ```bash -dotbot fw fetch # pull the latest pre-compiled firmwares from a release +dotbot fw fetch # pull the pinned pre-compiled firmwares (swarmit + dotbot-firmware) dotbot device flash-mari-gateway -s 10 # flash the gateway dotbot device flash-swarmit-sandbox -s 77 # the sandbox firmware - do this on each dotbot ``` (`device flash-mari-gateway` / `flash-swarmit-sandbox` auto-fetch -the release into `./artifacts/` if it isn't already there.) +the firmware into `~/.dotbot/artifacts/` if it isn't already there.) Now, run the gateway (the broker comes from your config): @@ -114,17 +114,18 @@ dotbot run gateway -p /dev/cu.usbmodem0010500324491 You can flash as many dotbots as you want, all at once! First, how about making them spinnnn 🔄 🔄 ```bash -dotbot swarm flash ./artifacts/spin-sandbox-dotbot-v3.bin -ys # flash the whole fleet with a simple spinning app +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-*/spin-sandbox-dotbot-v3.bin -ys # flash the whole fleet with a simple spinning app ``` (`dotbot swarm` reads the same `dotbot.toml` as the rest - pass `--conn` / -`--swarm-id` to override it for one run.) +`--swarm-id` to override it for one run. The `dotbot-firmware-*` glob expands to +the release `dotbot fw fetch` cached; run `dotbot fw list` for exact paths.) Then, flash another experiment: ```bash dotbot swarm stop # ensure all robots are in bootloader -dotbot swarm flash ./artifacts/dotbot-sandbox-dotbot-v3.bin -ys # this firmware allows bots to be remote-controlled +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-*/dotbot-sandbox-dotbot-v3.bin -ys # this firmware allows bots to be remote-controlled ``` Observe and control your swarm from a web interface: From dd45ef8470911406934779bd24e0d812b666aeb9 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:38:26 +0200 Subject: [PATCH 19/27] dotbot/cli: default device role flash to the pinned swarmit version Aligns the no-`-f` default with `dotbot fw fetch` so both stage the same `swarmit-/` cache dir; pass `-f latest` for the old behavior. AI-assisted: Claude Opus 4.8 --- dotbot/cli/device.py | 20 ++++++++++++-------- dotbot/tests/test_device.py | 19 +++++++++++++++++++ 2 files changed, 31 insertions(+), 8 deletions(-) diff --git a/dotbot/cli/device.py b/dotbot/cli/device.py index 22d7603e..e0c4977b 100644 --- a/dotbot/cli/device.py +++ b/dotbot/cli/device.py @@ -99,8 +99,8 @@ def _fw_version_option(f): "-f", default=None, help=( - "Release version to flash, e.g. 0.8.0rc2 (default: latest swarmit " - f"release). Binaries are fetched into {DEFAULT_ARTIFACTS_DISPLAY}/ if not cached." + "Release version to flash, e.g. 0.8.0rc2 (default: the swarmit " + f"version pydotbot pins). Binaries are fetched into {DEFAULT_ARTIFACTS_DISPLAY}/ if not cached." ), )(f) @@ -139,7 +139,7 @@ def flash_swarmit_sandbox( from dotbot.firmware.flash import ( flash_role, normalize_network_id, - resolve_latest_version, + pinned_version, ) swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) @@ -149,8 +149,10 @@ def flash_swarmit_sandbox( "deployment) in your config." ) if fw_version is None: - fw_version = resolve_latest_version() - click.echo(f"No version specified, using the latest release: {fw_version}") + fw_version = pinned_version("swarmit") + click.echo( + f"No version specified, using the pinned swarmit version: {fw_version}" + ) ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( @@ -182,7 +184,7 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): from dotbot.firmware.flash import ( flash_role, normalize_network_id, - resolve_latest_version, + pinned_version, ) swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) @@ -192,8 +194,10 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): "deployment) in your config." ) if fw_version is None: - fw_version = resolve_latest_version() - click.echo(f"No version specified, using the latest release: {fw_version}") + fw_version = pinned_version("swarmit") + click.echo( + f"No version specified, using the pinned swarmit version: {fw_version}" + ) ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index d0c76f9d..4569f6fd 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -100,6 +100,25 @@ def fake_flash_role(role, **kw): assert calls["kw"]["sn_starting_digits"] == "77" +def test_flash_swarmit_sandbox_defaults_to_pinned_version( + runner, _no_nrfjprog_gate, monkeypatch +): + """With no -f, the role flash uses the pinned swarmit version (matching + `fw fetch`), not the latest release - and resolves it without the network.""" + import dotbot.firmware.flash as flash + + calls = {} + monkeypatch.setattr( + "dotbot.firmware.flash.flash_role", + lambda role, **kw: calls.update(role=role, kw=kw), + ) + result = runner.invoke( + device_cmd, ["flash-swarmit-sandbox", "--swarm-id", "0100", "-s", "77"] + ) + assert result.exit_code == 0, result.output + assert calls["kw"]["fw_version"] == flash.pinned_version("swarmit") + + def test_flash_mari_gateway_calls_engine_with_gateway_role( runner, _no_nrfjprog_gate, monkeypatch ): From 406654d54e7d268003ffe20f41864f26b1b462b9 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:38:26 +0200 Subject: [PATCH 20/27] dotbot/config: drop the unwired artifacts_dir key The key was never read - $DOTBOT_ARTIFACTS_DIR is the cache-location override. A config that still sets it now fails (extra keys are forbidden). AI-assisted: Claude Opus 4.8 --- dotbot/config.py | 1 - dotbot/tests/test_cli_helpers.py | 3 +-- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/dotbot/config.py b/dotbot/config.py index 9e76a20b..250d048a 100644 --- a/dotbot/config.py +++ b/dotbot/config.py @@ -163,7 +163,6 @@ class DotbotConfig(_Strict): """The whole file: top-level shared keys + the four section tables + deployments.""" default_deployment: str | None = None - artifacts_dir: str | None = None log_level: str | None = None conn: Conn = None swarm_id: str | None = None diff --git a/dotbot/tests/test_cli_helpers.py b/dotbot/tests/test_cli_helpers.py index fe0968f4..c7f271eb 100644 --- a/dotbot/tests/test_cli_helpers.py +++ b/dotbot/tests/test_cli_helpers.py @@ -87,8 +87,7 @@ def test_config_show_skips_none_values(runner, tmp_path): cfg = _write(tmp_path, 'swarm_id = "0001"\n') result = runner.invoke(cli, ["-c", str(cfg), "config", "show"]) assert result.exit_code == 0, result.output - # `artifacts_dir`/`log_level` are unset (None) and must not appear. - assert "artifacts_dir" not in result.output + # `log_level` is unset (None) and must not appear. assert "log_level" not in result.output From 6307f553254c70bc9413233f4590467cdf36389b Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:38:26 +0200 Subject: [PATCH 21/27] doc: align fetch/flash docs with the pinned default and drop artifacts_dir AI-assisted: Claude Opus 4.8 --- doc/cli/device.md | 2 +- doc/cli/swarm.md | 2 +- doc/reference/configuration.md | 2 -- 3 files changed, 2 insertions(+), 4 deletions(-) diff --git a/doc/cli/device.md b/doc/cli/device.md index 4cf5fc9a..ca1f94ab 100644 --- a/doc/cli/device.md +++ b/doc/cli/device.md @@ -100,7 +100,7 @@ dotbot device flash-swarmit-sandbox --swarm-id 0100 -f 0.8.0rc1 -s 77 | Flag | `flash-mari-gateway` | `flash-swarmit-sandbox` | |---|---|---| | `--swarm-id` | 16-bit hex swarm id (or from config) | 16-bit hex swarm id (or from config) | -| `-f, --fw-version` | release to flash (required) | release to flash (required) | +| `-f, --fw-version` | release to flash (default: the pinned swarmit version) | release to flash (default: the pinned swarmit version) | | `-s, --sn-starting-digits` | J-Link serial prefix | J-Link serial prefix | | `-l, --calibration` | - | optional LH2 calibration file to bake in | diff --git a/doc/cli/swarm.md b/doc/cli/swarm.md index 17d47ccd..fb42e3b3 100644 --- a/doc/cli/swarm.md +++ b/doc/cli/swarm.md @@ -83,7 +83,7 @@ to override). If the broker needs auth, set `DOTBOT_MQTT_USER` / ```bash dotbot swarm status # who's out there + their state dotbot swarm status -w # keep watching -dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-*/spin-sandbox-dotbot-v3.bin -ys +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-local/spin-sandbox-dotbot-v3.bin -ys dotbot swarm stop # back to bootloader (before re-flashing) dotbot swarm start # (re)start the loaded app dotbot swarm monitor # tail SWARMIT_EVENT_LOG from bots diff --git a/doc/reference/configuration.md b/doc/reference/configuration.md index 6c5035e4..ead8a3ea 100644 --- a/doc/reference/configuration.md +++ b/doc/reference/configuration.md @@ -81,7 +81,6 @@ Set once at the top of the file; any section or deployment can override them. | `conn` | Default connection string (`mqtts://host:port`, a serial path, or `simulator`). | | `swarm_id` | Swarm id selecting the MQTT topic namespace. | | `log_level` | Logging verbosity. | -| `artifacts_dir` | Where firmware artifacts are read/written. | | `default_deployment` | Name of the deployment to select when neither `--deployment` nor `DOTBOT_DEPLOYMENT` is given. | ## Section tables @@ -220,7 +219,6 @@ default_deployment = "inria" # used when --deployment / DOTBOT_D conn = "mqtts://broker.local:8883" swarm_id = "0001" log_level = "info" -artifacts_dir = "~/.dotbot/artifacts" # A physical deployment. Select it with `--deployment inria`, DOTBOT_DEPLOYMENT, or # default_deployment above - don't edit this table to switch deployments. From 29f1e1a1b24bb8478726cedd008a7bae33ef0c9b Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:38:26 +0200 Subject: [PATCH 22/27] readme: show concrete swarm-flash paths in the quickstart AI-assisted: Claude Opus 4.8 --- README.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 240b8b83..f1fdfd67 100644 --- a/README.md +++ b/README.md @@ -114,18 +114,19 @@ dotbot run gateway -p /dev/cu.usbmodem0010500324491 You can flash as many dotbots as you want, all at once! First, how about making them spinnnn 🔄 🔄 ```bash -dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-*/spin-sandbox-dotbot-v3.bin -ys # flash the whole fleet with a simple spinning app +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-1.22.0rc1/spin-sandbox-dotbot-v3.bin -ys # flash the whole fleet with a simple spinning app ``` (`dotbot swarm` reads the same `dotbot.toml` as the rest - pass `--conn` / -`--swarm-id` to override it for one run. The `dotbot-firmware-*` glob expands to -the release `dotbot fw fetch` cached; run `dotbot fw list` for exact paths.) +`--swarm-id` to override it for one run. That path is the `dotbot-firmware` +release `dotbot fw fetch` cached - run `dotbot fw list` to see the exact paths +and versions on your machine.) Then, flash another experiment: ```bash dotbot swarm stop # ensure all robots are in bootloader -dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-*/dotbot-sandbox-dotbot-v3.bin -ys # this firmware allows bots to be remote-controlled +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-1.22.0rc1/dotbot-sandbox-dotbot-v3.bin -ys # this firmware allows bots to be remote-controlled ``` Observe and control your swarm from a web interface: From dfb25ad128452ef8dfd283174b04283c13c97a1a Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 15:45:57 +0200 Subject: [PATCH 23/27] dotbot/firmware: make _short_path tolerate Windows cross-drive paths AI-assisted: Claude Opus 4.8 --- dotbot/firmware/flash.py | 6 +++++- dotbot/tests/test_device.py | 14 ++++++++++++++ 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 0004a478..02e1e736 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -147,7 +147,11 @@ def _human_size(num_bytes: int) -> str: def _short_path(path: Path) -> str: """Path relative to the cwd when that's shorter, else absolute.""" - rel = os.path.relpath(path) + try: + rel = os.path.relpath(path) + except ValueError: + # Windows: path and cwd on different drives have no relative form. + return str(path) return rel if not rel.startswith("..") else str(path) diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index 4569f6fd..c664ba97 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -474,6 +474,20 @@ def not_found(url): assert sleeps == [] # never retried +def test_short_path_falls_back_to_absolute_across_drives(monkeypatch): + """On Windows os.path.relpath raises ValueError when the path and cwd are + on different drives (C: vs D:); _short_path must return the absolute path, + not crash.""" + import dotbot.firmware.flash as flash + + def boom(_p): + raise ValueError("path is on mount 'C:', start on mount 'D:'") + + monkeypatch.setattr(flash.os.path, "relpath", boom) + p = Path("/x/swarmit-1.2.3") + assert flash._short_path(p) == str(p) + + def test_pinned_version_dotbot_firmware_is_declared(): """DotBot-firmware (not a Python dep) pins to the declared constant.""" import dotbot.firmware.flash as flash From 4c9f223c526c619382ed964ede6676d2dd8c5d49 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 16:07:09 +0200 Subject: [PATCH 24/27] dotbot/firmware: split release fetching out of flash.py into fetch.py Pure move: flash.py keeps the device-flashing engine and imports fetch_assets/resolve_fw_root back from fetch.py for flash_role's auto-fetch (one-way dep, no cycle). No behavior change. AI-assisted: Claude Opus 4.8 --- dotbot/cli/device.py | 14 +- dotbot/cli/fw.py | 2 +- dotbot/firmware/fetch.py | 302 ++++++++++++++++++++++++++++++++++++ dotbot/firmware/flash.py | 295 +---------------------------------- dotbot/tests/test_device.py | 80 ++++------ 5 files changed, 345 insertions(+), 348 deletions(-) create mode 100644 dotbot/firmware/fetch.py diff --git a/dotbot/cli/device.py b/dotbot/cli/device.py index e0c4977b..f0b92323 100644 --- a/dotbot/cli/device.py +++ b/dotbot/cli/device.py @@ -136,11 +136,8 @@ def flash_swarmit_sandbox( network identity. Auto-fetches the release if not already in ~/.dotbot/artifacts/swarmit-/. """ - from dotbot.firmware.flash import ( - flash_role, - normalize_network_id, - pinned_version, - ) + from dotbot.firmware.fetch import pinned_version + from dotbot.firmware.flash import flash_role, normalize_network_id swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) if swarm_id is None: @@ -181,11 +178,8 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): identity. Auto-fetches the release if absent. (To run the host-side UART<->MQTT bridge instead, use `dotbot run gateway`.) """ - from dotbot.firmware.flash import ( - flash_role, - normalize_network_id, - pinned_version, - ) + from dotbot.firmware.fetch import pinned_version + from dotbot.firmware.flash import flash_role, normalize_network_id swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) if swarm_id is None: diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index 6e47395f..a1d4966e 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -306,7 +306,7 @@ def fetch(source, fw_version, local_root): release or `-f ` for a specific one. """ from dotbot import pydotbot_version - from dotbot.firmware.flash import ( + from dotbot.firmware.fetch import ( DEFAULT_FETCH_SOURCES, _short_path, fetch_assets, diff --git a/dotbot/firmware/fetch.py b/dotbot/firmware/fetch.py new file mode 100644 index 00000000..6c13eca3 --- /dev/null +++ b/dotbot/firmware/fetch.py @@ -0,0 +1,302 @@ +"""Firmware release fetching + cache layout (no CLI, no flashing). + +The engine behind `dotbot fw fetch` and the auto-fetch hook in +`flash.flash_role`: decide which GitHub release to pull (pinned / latest / +an explicit tag), download every asset it publishes into the +source-qualified cache (`~/.dotbot/artifacts/-/`), and +record provenance in a `manifest.json`. Pure library code; the Click +surface lives in `dotbot/cli/fw.py`. + +Kept separate from `flash.py` (the hardware-facing flashing engine): this +module never touches a device, only the network + the cache. +""" + +from __future__ import annotations + +import json +import os +import shutil +import time +import urllib.error +import urllib.request +from concurrent.futures import ThreadPoolExecutor, as_completed +from datetime import datetime, timezone +from pathlib import Path + +import click + +GITHUB_API = "https://api.github.com/repos" +# Firmware release sources. swarmit ships the swarm system images (bootloader +# + mari netcore + mari gateway); DotBot-firmware ships the bare apps (.hex) +# and the sandbox apps (.bin). Each is cached in its own -/ +# subdir of the artifacts cache so versions and provenance never collide. +RELEASE_SOURCES = { + "swarmit": "DotBots/swarmit", + "dotbot-firmware": "DotBots/DotBot-firmware", +} +DEFAULT_FETCH_SOURCES = ("swarmit", "dotbot-firmware") +# The DotBot-firmware release this pydotbot is built and tested against. Unlike +# swarmit (a Python dependency, whose firmware release is tagged identically to +# the package, so we read it from importlib.metadata), DotBot-firmware is not a +# Python package - so the expected version is declared here and bumped +# deliberately when pydotbot adopts a new release. `dotbot fw fetch` (no -f) +# pulls exactly this; -f overrides it. +DOTBOT_FIRMWARE_VERSION = "1.22.0rc1" + +# Transient HTTP statuses worth retrying (GitHub's asset CDN 502s now and +# then under concurrent load; 429 is rate-limiting). +_RETRY_STATUS = {429, 500, 502, 503, 504} + + +def resolve_fw_root(bin_dir: Path, source: str, fw_version: str) -> Path: + return bin_dir / f"{source}-{fw_version}" + + +def _human_size(num_bytes: int) -> str: + size = float(num_bytes) + for unit in ("B", "KB", "MB", "GB"): + if size < 1024 or unit == "GB": + return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}" + size /= 1024 + + +def _short_path(path: Path) -> str: + """Path relative to the cwd when that's shorter, else absolute.""" + try: + rel = os.path.relpath(path) + except ValueError: + # Windows: path and cwd on different drives have no relative form. + return str(path) + return rel if not rel.startswith("..") else str(path) + + +def download_file(url: str, dest: Path, *, retries: int = 3) -> int: + """Download ``url`` to ``dest``; return the number of bytes written. + + Retries transient failures (connection errors and HTTP 429/5xx) with + exponential backoff - GitHub's CDN occasionally 502s under concurrent + downloads, and a sporadic failure shouldn't abort the whole fetch. + """ + for attempt in range(retries + 1): + try: + with urllib.request.urlopen(url) as resp: + status = getattr(resp, "status", 200) + if status != 200: + raise click.ClickException(f"HTTP {status} while downloading {url}") + data = resp.read() + dest.write_bytes(data) + return len(data) + except urllib.error.HTTPError as exc: + if exc.code not in _RETRY_STATUS or attempt == retries: + raise click.ClickException( + f"HTTP error while downloading {url}: {exc}" + ) from exc + reason = f"HTTP {exc.code}" + except urllib.error.URLError as exc: + if attempt == retries: + raise click.ClickException( + f"Network error while downloading {url}: {exc}" + ) from exc + reason = str(exc.reason) + delay = 0.5 * (2**attempt) + click.echo( + f" [retry] {url.rsplit('/', 1)[-1]} ({reason}); retrying in {delay:.1f}s", + err=True, + ) + time.sleep(delay) + raise click.ClickException( # pragma: no cover - loop always returns/raises + f"Failed to download {url} after {retries} retries." + ) + + +def _github_get(url: str): + """Unauthenticated GitHub API GET (60 req/hour is plenty for a CLI).""" + request = urllib.request.Request( + url, + headers={"Accept": "application/vnd.github+json", "User-Agent": "dotbot"}, + ) + try: + with urllib.request.urlopen(request) as resp: + return json.load(resp) + except (urllib.error.HTTPError, urllib.error.URLError) as exc: + raise click.ClickException(f"GitHub API request failed ({url}): {exc}") from exc + + +def resolve_release(source: str, fw_version: str) -> dict: + """Return the GitHub release JSON for ``source`` at ``fw_version``. + + ``fw_version="latest"`` resolves the newest release via ``/releases`` + (prereleases included, unlike ``/releases/latest`` which skips them - the + current newest, e.g. swarmit 0.8.0rc2, is a prerelease). + """ + if source not in RELEASE_SOURCES: + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + repo = RELEASE_SOURCES[source] + if fw_version == "latest": + releases = _github_get(f"{GITHUB_API}/{repo}/releases?per_page=1") + if not releases: + raise click.ClickException( + f"No releases found for {repo}; pass an explicit -f ." + ) + return releases[0] + return _github_get(f"{GITHUB_API}/{repo}/releases/tags/{fw_version}") + + +def resolve_latest_version(source: str = "swarmit") -> str: + """The newest release tag for ``source`` (prereleases included).""" + return resolve_release(source, "latest")["tag_name"] + + +def pinned_version(source: str) -> str: + """The exact firmware release this pydotbot pins for ``source``. + + swarmit is a Python dependency whose firmware release is tagged identically + to the package, so its pin is read from the installed package. DotBot-firmware + is not a Python package, so its pin is the declared ``DOTBOT_FIRMWARE_VERSION``. + `dotbot fw fetch` (no -f) resolves to these; pass -f to override. + """ + if source == "swarmit": + from importlib.metadata import PackageNotFoundError + from importlib.metadata import version as _pkg_version + + try: + return _pkg_version("swarmit") + except PackageNotFoundError as exc: + raise click.ClickException( + "Cannot infer the swarmit firmware version: the swarmit package " + "is not installed. Pass -f explicitly." + ) from exc + if source == "dotbot-firmware": + return DOTBOT_FIRMWARE_VERSION + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + + +def fetch_assets( + source: str, fw_version: str, bin_dir: Path, local_root: Path | None = None +) -> Path: + """Fetch one source's firmware into ``bin_dir/-/``. + + For a released version, downloads every ``.hex``/``.bin`` asset the GitHub + release publishes (so it adapts to whatever the release ships, no hardcoded + asset list) and writes a ``manifest.json`` with provenance. For + ``fw_version="local"``, symlinks/copies from a local build tree into + ``-local/``. Used by `dotbot fw fetch` and the auto-fetch hook in + `flash_role`. + """ + if source not in RELEASE_SOURCES: + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + if fw_version == "local" and not local_root: + raise click.ClickException("--local-root is required when --fw-version=local.") + if fw_version != "local" and local_root: + click.echo( + "[WARN] --local-root ignored when --fw-version is not 'local'.", + err=True, + ) + + if fw_version == "local": + return _link_local_assets(source, local_root, bin_dir) + + release = resolve_release(source, fw_version) + tag = release["tag_name"] + out_dir = resolve_fw_root(bin_dir, source, tag) + out_dir.mkdir(parents=True, exist_ok=True) + assets = [ + a for a in release.get("assets", []) if a["name"].endswith((".hex", ".bin")) + ] + if not assets: + raise click.ClickException( + f"{source} release {tag} publishes no .hex/.bin assets." + ) + click.echo( + f"Fetching {source} {tag} ({len(assets)} files) into {_short_path(out_dir)}" + ) + names: list[str] = [] + errors: list[str] = [] + # Downloads are I/O-bound, so a small thread pool overlaps them (urllib + # releases the GIL during the network read). Sources stay sequential. + # Kept modest: GitHub's asset CDN starts 502ing under heavier fan-out + # (download_file retries transient failures regardless). + with ThreadPoolExecutor(max_workers=4) as pool: + future_to_name = { + pool.submit( + download_file, asset["browser_download_url"], out_dir / asset["name"] + ): asset["name"] + for asset in assets + } + for future in as_completed(future_to_name): + name = future_to_name[future] + try: + size = future.result() + except click.ClickException as exc: + errors.append(f"{name}: {exc.format_message()}") + continue + names.append(name) + click.echo(f" {name} ({_human_size(size)})") + if errors: + raise click.ClickException( + f"{len(errors)} asset(s) failed to download:\n " + "\n ".join(errors) + ) + _write_manifest(out_dir, source, tag, RELEASE_SOURCES[source], names) + return out_dir + + +def _write_manifest( + out_dir: Path, source: str, version: str, repo: str, files: list[str] +) -> None: + """Record provenance next to the binaries (a cheap audit trail).""" + from dotbot import pydotbot_version + + manifest = { + "source": source, + "version": version, + "repo": repo, + "url": f"https://github.com/{repo}/releases/tag/{version}", + "pydotbot": pydotbot_version(), + "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "files": sorted(files), + } + (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") + + +def _link_local_assets(source: str, local_root: Path, bin_dir: Path) -> Path: + """Symlink/copy a local build tree into ``bin_dir/-local/``.""" + if source != "swarmit": + raise click.ClickException( + f"--fw-version local is only wired for source 'swarmit' so far, " + f"not '{source}'." + ) + local_root = local_root.expanduser().resolve() + out_dir = resolve_fw_root(bin_dir, source, "local") + out_dir.mkdir(parents=True, exist_ok=True) + mapping = { + "bootloader-dotbot-v3.hex": local_root + / "device/bootloader/Output/dotbot-v3/Debug/Exe/bootloader-dotbot-v3.hex", + "netcore-nrf5340-net.hex": local_root + / "device/network_core/Output/nrf5340-net/Debug/Exe/netcore-nrf5340-net.hex", + "03app_gateway_app-nrf5340-app.hex": local_root + / "mari/firmware/app/03app_gateway_app/Output/nrf5340-app/Debug/Exe/03app_gateway_app-nrf5340-app.hex", + "03app_gateway_net-nrf5340-net.hex": local_root + / "mari/firmware/app/03app_gateway_net/Output/nrf5340-net/Debug/Exe/03app_gateway_net-nrf5340-net.hex", + } + missing = [name for name, src in mapping.items() if not src.exists()] + if missing: + raise click.ClickException( + f"Missing local build artifacts: {', '.join(missing)}" + ) + for name, src in mapping.items(): + dest = out_dir / name + if dest.exists() or dest.is_symlink(): + dest.unlink() + try: + os.symlink(src, dest) + click.echo(f"[LINK] {dest} -> {src}") + except OSError: + shutil.copy2(src, dest) + click.echo(f"[COPY] {dest} <- {src}") + return out_dir diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 02e1e736..943592dc 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -1,28 +1,23 @@ """DotBot firmware flashing + provisioning engine (no CLI). -The hardware-facing engine behind `dotbot device` and `dotbot fw fetch`: -fetch release assets, flash a role's system firmware + config page -(`flash_role`), flash a single app image (`flash_app_image`), flash the -debug-chip programmer (`flash_programmer`), and read back provisioning -state (`read_config_report`). The Click surface lives in -`dotbot/cli/device.py` and `dotbot/cli/fw.py`; this module is pure -library code. +The hardware-facing engine behind `dotbot device`: flash a role's system +firmware + config page (`flash_role`), flash a single app image +(`flash_app_image`), flash the debug-chip programmer (`flash_programmer`), +and read back provisioning state (`read_config_report`). Release fetching ++ the artifact cache live next door in `fetch.py`; the Click surface lives +in `dotbot/cli/device.py`. This module is pure library code. """ from __future__ import annotations import json import os -import shutil import time -import urllib.error -import urllib.request -from concurrent.futures import ThreadPoolExecutor, as_completed -from datetime import datetime, timezone from pathlib import Path import click +from .fetch import fetch_assets, resolve_fw_root from .nrf import ( do_daplink, do_daplink_if, @@ -56,23 +51,6 @@ # dotbot-lh2-calibration (1-byte count + N matrices of 3x3 int32 LE). LH2_MATRIX_BYTES = 3 * 3 * 4 # 3x3 int32 matrix LH2_MAX_HOMOGRAPHIES = 16 -GITHUB_API = "https://api.github.com/repos" -# Firmware release sources. swarmit ships the swarm system images (bootloader -# + mari netcore + mari gateway); DotBot-firmware ships the bare apps (.hex) -# and the sandbox apps (.bin). Each is cached in its own -/ -# subdir of the artifacts cache so versions and provenance never collide. -RELEASE_SOURCES = { - "swarmit": "DotBots/swarmit", - "dotbot-firmware": "DotBots/DotBot-firmware", -} -DEFAULT_FETCH_SOURCES = ("swarmit", "dotbot-firmware") -# The DotBot-firmware release this pydotbot is built and tested against. Unlike -# swarmit (a Python dependency, whose firmware release is tagged identically to -# the package, so we read it from importlib.metadata), DotBot-firmware is not a -# Python package - so the expected version is declared here and bumped -# deliberately when pydotbot adopts a new release. `dotbot fw fetch` (no -f) -# pulls exactly this; -f overrides it. -DOTBOT_FIRMWARE_VERSION = "1.22.0rc1" # Application images are linked after the bootloader. APP_FLASH_BASE_ADDR = 0x00010000 # Programmer bring-up files @@ -133,138 +111,6 @@ def normalize_network_id(raw: str | None) -> tuple[int, str] | None: return value, f"{value:04X}" -def resolve_fw_root(bin_dir: Path, source: str, fw_version: str) -> Path: - return bin_dir / f"{source}-{fw_version}" - - -def _human_size(num_bytes: int) -> str: - size = float(num_bytes) - for unit in ("B", "KB", "MB", "GB"): - if size < 1024 or unit == "GB": - return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}" - size /= 1024 - - -def _short_path(path: Path) -> str: - """Path relative to the cwd when that's shorter, else absolute.""" - try: - rel = os.path.relpath(path) - except ValueError: - # Windows: path and cwd on different drives have no relative form. - return str(path) - return rel if not rel.startswith("..") else str(path) - - -# Transient HTTP statuses worth retrying (GitHub's asset CDN 502s now and -# then under concurrent load; 429 is rate-limiting). -_RETRY_STATUS = {429, 500, 502, 503, 504} - - -def download_file(url: str, dest: Path, *, retries: int = 3) -> int: - """Download ``url`` to ``dest``; return the number of bytes written. - - Retries transient failures (connection errors and HTTP 429/5xx) with - exponential backoff - GitHub's CDN occasionally 502s under concurrent - downloads, and a sporadic failure shouldn't abort the whole fetch. - """ - for attempt in range(retries + 1): - try: - with urllib.request.urlopen(url) as resp: - status = getattr(resp, "status", 200) - if status != 200: - raise click.ClickException(f"HTTP {status} while downloading {url}") - data = resp.read() - dest.write_bytes(data) - return len(data) - except urllib.error.HTTPError as exc: - if exc.code not in _RETRY_STATUS or attempt == retries: - raise click.ClickException( - f"HTTP error while downloading {url}: {exc}" - ) from exc - reason = f"HTTP {exc.code}" - except urllib.error.URLError as exc: - if attempt == retries: - raise click.ClickException( - f"Network error while downloading {url}: {exc}" - ) from exc - reason = str(exc.reason) - delay = 0.5 * (2**attempt) - click.echo( - f" [retry] {url.rsplit('/', 1)[-1]} ({reason}); retrying in {delay:.1f}s", - err=True, - ) - time.sleep(delay) - raise click.ClickException( # pragma: no cover - loop always returns/raises - f"Failed to download {url} after {retries} retries." - ) - - -def _github_get(url: str): - """Unauthenticated GitHub API GET (60 req/hour is plenty for a CLI).""" - request = urllib.request.Request( - url, - headers={"Accept": "application/vnd.github+json", "User-Agent": "dotbot"}, - ) - try: - with urllib.request.urlopen(request) as resp: - return json.load(resp) - except (urllib.error.HTTPError, urllib.error.URLError) as exc: - raise click.ClickException(f"GitHub API request failed ({url}): {exc}") from exc - - -def resolve_release(source: str, fw_version: str) -> dict: - """Return the GitHub release JSON for ``source`` at ``fw_version``. - - ``fw_version="latest"`` resolves the newest release via ``/releases`` - (prereleases included, unlike ``/releases/latest`` which skips them - the - current newest, e.g. swarmit 0.8.0rc2, is a prerelease). - """ - if source not in RELEASE_SOURCES: - raise click.ClickException( - f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." - ) - repo = RELEASE_SOURCES[source] - if fw_version == "latest": - releases = _github_get(f"{GITHUB_API}/{repo}/releases?per_page=1") - if not releases: - raise click.ClickException( - f"No releases found for {repo}; pass an explicit -f ." - ) - return releases[0] - return _github_get(f"{GITHUB_API}/{repo}/releases/tags/{fw_version}") - - -def resolve_latest_version(source: str = "swarmit") -> str: - """The newest release tag for ``source`` (prereleases included).""" - return resolve_release(source, "latest")["tag_name"] - - -def pinned_version(source: str) -> str: - """The exact firmware release this pydotbot pins for ``source``. - - swarmit is a Python dependency whose firmware release is tagged identically - to the package, so its pin is read from the installed package. DotBot-firmware - is not a Python package, so its pin is the declared ``DOTBOT_FIRMWARE_VERSION``. - `dotbot fw fetch` (no -f) resolves to these; pass -f to override. - """ - if source == "swarmit": - from importlib.metadata import PackageNotFoundError - from importlib.metadata import version as _pkg_version - - try: - return _pkg_version("swarmit") - except PackageNotFoundError as exc: - raise click.ClickException( - "Cannot infer the swarmit firmware version: the swarmit package " - "is not installed. Pass -f explicitly." - ) from exc - if source == "dotbot-firmware": - return DOTBOT_FIRMWARE_VERSION - raise click.ClickException( - f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." - ) - - def convert_bin_to_hex(bin_path: Path, base_addr: int) -> Path: if IntelHex is None: raise click.ClickException( @@ -451,133 +297,6 @@ def manifest_matches( ) -def fetch_assets( - source: str, fw_version: str, bin_dir: Path, local_root: Path | None = None -) -> Path: - """Fetch one source's firmware into ``bin_dir/-/``. - - For a released version, downloads every ``.hex``/``.bin`` asset the GitHub - release publishes (so it adapts to whatever the release ships, no hardcoded - asset list) and writes a ``manifest.json`` with provenance. For - ``fw_version="local"``, symlinks/copies from a local build tree into - ``-local/``. Used by `dotbot fw fetch` and the auto-fetch hook in - `flash_role`. - """ - if source not in RELEASE_SOURCES: - raise click.ClickException( - f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." - ) - if fw_version == "local" and not local_root: - raise click.ClickException("--local-root is required when --fw-version=local.") - if fw_version != "local" and local_root: - click.echo( - "[WARN] --local-root ignored when --fw-version is not 'local'.", - err=True, - ) - - if fw_version == "local": - return _link_local_assets(source, local_root, bin_dir) - - release = resolve_release(source, fw_version) - tag = release["tag_name"] - out_dir = resolve_fw_root(bin_dir, source, tag) - out_dir.mkdir(parents=True, exist_ok=True) - assets = [ - a for a in release.get("assets", []) if a["name"].endswith((".hex", ".bin")) - ] - if not assets: - raise click.ClickException( - f"{source} release {tag} publishes no .hex/.bin assets." - ) - click.echo( - f"Fetching {source} {tag} ({len(assets)} files) into {_short_path(out_dir)}" - ) - names: list[str] = [] - errors: list[str] = [] - # Downloads are I/O-bound, so a small thread pool overlaps them (urllib - # releases the GIL during the network read). Sources stay sequential. - # Kept modest: GitHub's asset CDN starts 502ing under heavier fan-out - # (download_file retries transient failures regardless). - with ThreadPoolExecutor(max_workers=4) as pool: - future_to_name = { - pool.submit( - download_file, asset["browser_download_url"], out_dir / asset["name"] - ): asset["name"] - for asset in assets - } - for future in as_completed(future_to_name): - name = future_to_name[future] - try: - size = future.result() - except click.ClickException as exc: - errors.append(f"{name}: {exc.format_message()}") - continue - names.append(name) - click.echo(f" {name} ({_human_size(size)})") - if errors: - raise click.ClickException( - f"{len(errors)} asset(s) failed to download:\n " + "\n ".join(errors) - ) - _write_manifest(out_dir, source, tag, RELEASE_SOURCES[source], names) - return out_dir - - -def _write_manifest( - out_dir: Path, source: str, version: str, repo: str, files: list[str] -) -> None: - """Record provenance next to the binaries (a cheap audit trail).""" - from dotbot import pydotbot_version - - manifest = { - "source": source, - "version": version, - "repo": repo, - "url": f"https://github.com/{repo}/releases/tag/{version}", - "pydotbot": pydotbot_version(), - "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), - "files": sorted(files), - } - (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") - - -def _link_local_assets(source: str, local_root: Path, bin_dir: Path) -> Path: - """Symlink/copy a local build tree into ``bin_dir/-local/``.""" - if source != "swarmit": - raise click.ClickException( - f"--fw-version local is only wired for source 'swarmit' so far, " - f"not '{source}'." - ) - local_root = local_root.expanduser().resolve() - out_dir = resolve_fw_root(bin_dir, source, "local") - out_dir.mkdir(parents=True, exist_ok=True) - mapping = { - "bootloader-dotbot-v3.hex": local_root - / "device/bootloader/Output/dotbot-v3/Debug/Exe/bootloader-dotbot-v3.hex", - "netcore-nrf5340-net.hex": local_root - / "device/network_core/Output/nrf5340-net/Debug/Exe/netcore-nrf5340-net.hex", - "03app_gateway_app-nrf5340-app.hex": local_root - / "mari/firmware/app/03app_gateway_app/Output/nrf5340-app/Debug/Exe/03app_gateway_app-nrf5340-app.hex", - "03app_gateway_net-nrf5340-net.hex": local_root - / "mari/firmware/app/03app_gateway_net/Output/nrf5340-net/Debug/Exe/03app_gateway_net-nrf5340-net.hex", - } - missing = [name for name, src in mapping.items() if not src.exists()] - if missing: - raise click.ClickException( - f"Missing local build artifacts: {', '.join(missing)}" - ) - for name, src in mapping.items(): - dest = out_dir / name - if dest.exists() or dest.is_symlink(): - dest.unlink() - try: - os.symlink(src, dest) - click.echo(f"[LINK] {dest} -> {src}") - except OSError: - shutil.copy2(src, dest) - click.echo(f"[COPY] {dest} <- {src}") - return out_dir - - def flash_role( role: str, *, diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index c664ba97..551176b2 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -16,6 +16,8 @@ import pytest from click.testing import CliRunner +import dotbot.firmware.fetch as fetch +import dotbot.firmware.flash as flash from dotbot.cli.device import _looks_like_path from dotbot.cli.device import cmd as device_cmd @@ -105,7 +107,6 @@ def test_flash_swarmit_sandbox_defaults_to_pinned_version( ): """With no -f, the role flash uses the pinned swarmit version (matching `fw fetch`), not the latest release - and resolves it without the network.""" - import dotbot.firmware.flash as flash calls = {} monkeypatch.setattr( @@ -116,7 +117,7 @@ def test_flash_swarmit_sandbox_defaults_to_pinned_version( device_cmd, ["flash-swarmit-sandbox", "--swarm-id", "0100", "-s", "77"] ) assert result.exit_code == 0, result.output - assert calls["kw"]["fw_version"] == flash.pinned_version("swarmit") + assert calls["kw"]["fw_version"] == fetch.pinned_version("swarmit") def test_flash_mari_gateway_calls_engine_with_gateway_role( @@ -349,7 +350,6 @@ def test_create_config_hex_appends_calibration(tmp_path): def test_intelhex_is_a_core_dependency(): """intelhex was folded into core deps (the [provision] extra is gone), so config-hex building works on a default `pip install pydotbot`.""" - import dotbot.firmware.flash as flash assert flash.IntelHex is not None @@ -359,8 +359,6 @@ def test_fetch_assets_downloads_release_into_source_version_dir(tmp_path, monkey -/, skips .elf/.map, and writes a manifest.""" import json as _json - import dotbot.firmware.flash as flash - fake_release = { "tag_name": "0.8.0rc2", "assets": [ @@ -369,14 +367,14 @@ def test_fetch_assets_downloads_release_into_source_version_dir(tmp_path, monkey {"name": "bootloader-dotbot-v3.elf", "browser_download_url": "u3"}, ], } - monkeypatch.setattr(flash, "resolve_release", lambda source, version: fake_release) + monkeypatch.setattr(fetch, "resolve_release", lambda source, version: fake_release) def fake_download(url, dest): dest.write_bytes(b"\x00") return 1 - monkeypatch.setattr(flash, "download_file", fake_download) - out = flash.fetch_assets("swarmit", "latest", tmp_path) + monkeypatch.setattr(fetch, "download_file", fake_download) + out = fetch.fetch_assets("swarmit", "latest", tmp_path) assert out == tmp_path / "swarmit-0.8.0rc2" assert (out / "bootloader-dotbot-v3.hex").exists() assert (out / "netcore-nrf5340-net.hex").exists() @@ -390,10 +388,9 @@ def fake_download(url, dest): def test_fetch_assets_unknown_source_errors(tmp_path): """An unknown source is a clear error, not a KeyError.""" - import dotbot.firmware.flash as flash with pytest.raises(click.ClickException): - flash.fetch_assets("not-a-source", "latest", tmp_path) + fetch.fetch_assets("not-a-source", "latest", tmp_path) def test_resolve_latest_version_returns_newest_tag(monkeypatch): @@ -401,36 +398,31 @@ def test_resolve_latest_version_returns_newest_tag(monkeypatch): import io import json - import dotbot.firmware.flash as flash - payload = json.dumps([{"tag_name": "0.8.0rc2"}, {"tag_name": "0.8.0rc1"}]).encode() monkeypatch.setattr( - flash.urllib.request, "urlopen", lambda req: io.BytesIO(payload) + fetch.urllib.request, "urlopen", lambda req: io.BytesIO(payload) ) - assert flash.resolve_latest_version() == "0.8.0rc2" + assert fetch.resolve_latest_version() == "0.8.0rc2" def test_resolve_latest_version_no_releases_errors(monkeypatch): """An empty release list is a clear error, not an IndexError.""" import io - import dotbot.firmware.flash as flash - - monkeypatch.setattr(flash.urllib.request, "urlopen", lambda req: io.BytesIO(b"[]")) + monkeypatch.setattr(fetch.urllib.request, "urlopen", lambda req: io.BytesIO(b"[]")) with pytest.raises(click.ClickException): - flash.resolve_latest_version() + fetch.resolve_latest_version() def test_resolve_latest_version_network_error_errors(monkeypatch): """A network failure surfaces as a friendly ClickException.""" - import dotbot.firmware.flash as flash def boom(req): - raise flash.urllib.error.URLError("offline") + raise fetch.urllib.error.URLError("offline") - monkeypatch.setattr(flash.urllib.request, "urlopen", boom) + monkeypatch.setattr(fetch.urllib.request, "urlopen", boom) with pytest.raises(click.ClickException): - flash.resolve_latest_version() + fetch.resolve_latest_version() def test_download_file_retries_transient_5xx(tmp_path, monkeypatch): @@ -438,21 +430,19 @@ def test_download_file_retries_transient_5xx(tmp_path, monkeypatch): succeeds - one bad gateway shouldn't abort the whole fetch.""" import io - import dotbot.firmware.flash as flash - calls = {"n": 0} def flaky_urlopen(url): calls["n"] += 1 if calls["n"] == 1: - raise flash.urllib.error.HTTPError(url, 502, "Bad Gateway", {}, None) + raise fetch.urllib.error.HTTPError(url, 502, "Bad Gateway", {}, None) return io.BytesIO(b"\xde\xad") - monkeypatch.setattr(flash.urllib.request, "urlopen", flaky_urlopen) - monkeypatch.setattr(flash.time, "sleep", lambda _delay: None) # skip real backoff + monkeypatch.setattr(fetch.urllib.request, "urlopen", flaky_urlopen) + monkeypatch.setattr(fetch.time, "sleep", lambda _delay: None) # skip real backoff dest = tmp_path / "spin-dotbot-v3.hex" - size = flash.download_file("http://x/spin-dotbot-v3.hex", dest, retries=3) + size = fetch.download_file("http://x/spin-dotbot-v3.hex", dest, retries=3) assert size == 2 assert dest.read_bytes() == b"\xde\xad" assert calls["n"] == 2 # one retry @@ -460,17 +450,16 @@ def flaky_urlopen(url): def test_download_file_gives_up_on_non_transient(tmp_path, monkeypatch): """A 404 is not transient - it surfaces immediately, with no backoff.""" - import dotbot.firmware.flash as flash def not_found(url): - raise flash.urllib.error.HTTPError(url, 404, "Not Found", {}, None) + raise fetch.urllib.error.HTTPError(url, 404, "Not Found", {}, None) sleeps: list[float] = [] - monkeypatch.setattr(flash.urllib.request, "urlopen", not_found) - monkeypatch.setattr(flash.time, "sleep", lambda d: sleeps.append(d)) + monkeypatch.setattr(fetch.urllib.request, "urlopen", not_found) + monkeypatch.setattr(fetch.time, "sleep", lambda d: sleeps.append(d)) with pytest.raises(click.ClickException): - flash.download_file("http://x/missing.hex", tmp_path / "missing.hex", retries=3) + fetch.download_file("http://x/missing.hex", tmp_path / "missing.hex", retries=3) assert sleeps == [] # never retried @@ -478,50 +467,44 @@ def test_short_path_falls_back_to_absolute_across_drives(monkeypatch): """On Windows os.path.relpath raises ValueError when the path and cwd are on different drives (C: vs D:); _short_path must return the absolute path, not crash.""" - import dotbot.firmware.flash as flash def boom(_p): raise ValueError("path is on mount 'C:', start on mount 'D:'") - monkeypatch.setattr(flash.os.path, "relpath", boom) + monkeypatch.setattr(fetch.os.path, "relpath", boom) p = Path("/x/swarmit-1.2.3") - assert flash._short_path(p) == str(p) + assert fetch._short_path(p) == str(p) def test_pinned_version_dotbot_firmware_is_declared(): """DotBot-firmware (not a Python dep) pins to the declared constant.""" - import dotbot.firmware.flash as flash - assert flash.pinned_version("dotbot-firmware") == flash.DOTBOT_FIRMWARE_VERSION + assert fetch.pinned_version("dotbot-firmware") == fetch.DOTBOT_FIRMWARE_VERSION def test_pinned_version_swarmit_from_installed_package(): """swarmit's firmware version is inferred from the installed package.""" import importlib.metadata as md - import dotbot.firmware.flash as flash - - assert flash.pinned_version("swarmit") == md.version("swarmit") + assert fetch.pinned_version("swarmit") == md.version("swarmit") def test_pinned_version_unknown_source_errors(): """An unknown source is a clear error, not a KeyError.""" - import dotbot.firmware.flash as flash with pytest.raises(click.ClickException): - flash.pinned_version("not-a-source") + fetch.pinned_version("not-a-source") def test_fetch_no_args_resolves_pinned_versions(monkeypatch): """`dotbot fw fetch` with no flags fetches the pinned version per source, not 'latest'.""" - import dotbot.firmware.flash as flash from dotbot.cli.fw import cmd as fw_cmd calls: list[tuple[str, str]] = [] - monkeypatch.setattr(flash, "pinned_version", lambda src: f"PIN-{src}") + monkeypatch.setattr(fetch, "pinned_version", lambda src: f"PIN-{src}") monkeypatch.setattr( - flash, + fetch, "fetch_assets", lambda src, version, bin_dir, local_root=None: ( calls.append((src, version)) or Path(f"/x/{src}-{version}") @@ -538,14 +521,13 @@ def test_fetch_no_args_resolves_pinned_versions(monkeypatch): def test_fetch_explicit_version_overrides_pin(monkeypatch): """-f with --source bypasses the pin and passes through verbatim.""" - import dotbot.firmware.flash as flash from dotbot.cli.fw import cmd as fw_cmd calls: list[tuple[str, str]] = [] pin_called: list[str] = [] - monkeypatch.setattr(flash, "pinned_version", lambda src: pin_called.append(src)) + monkeypatch.setattr(fetch, "pinned_version", lambda src: pin_called.append(src)) monkeypatch.setattr( - flash, + fetch, "fetch_assets", lambda src, version, bin_dir, local_root=None: ( calls.append((src, version)) or Path(f"/x/{src}-{version}") From cdc15ea1c17d723c5661a2791924f3c2174c6b68 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Wed, 3 Jun 2026 16:15:51 +0200 Subject: [PATCH 25/27] doc: skip the 403-ing dotbots.org + segger.com links in linkcheck AI-assisted: Claude Opus 4.8 --- doc/conf.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/conf.py b/doc/conf.py index 1c10a4a2..35b78e48 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -134,6 +134,10 @@ # intermittently time out or rate-limit the linkcheck bot. r"https://img\.shields\.io/", r"https://badge\.fury\.io/", + # dotbots.org and segger.com return 403 to the linkcheck bot (WAF / + # user-agent block); both links are valid for humans. + r"https?://www\.dotbots\.org", + r"https://www\.segger\.com/", ] # -- Options for autosummary/autodoc output ----------------------------------- From a215bb10a52989593484c853c975a41476bc0bac Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Thu, 4 Jun 2026 12:53:56 +0200 Subject: [PATCH 26/27] doc: lead LH2 calibration with the OTA flow, split out the cabled guide AI-assisted: Claude Opus 4.8 --- README.md | 45 ++++++---- doc/cli/fw.md | 2 +- doc/cli/run.md | 12 +-- doc/cli/swarm.md | 20 +++-- doc/guides/index.md | 5 +- doc/guides/lh2-calibration-cabled.md | 87 ++++++++++++++++++++ doc/guides/lh2-calibration.md | 118 +++++++++++++-------------- doc/guides/one-bot.md | 5 +- 8 files changed, 201 insertions(+), 93 deletions(-) create mode 100644 doc/guides/lh2-calibration-cabled.md diff --git a/README.md b/README.md index f1fdfd67..27364e0c 100644 --- a/README.md +++ b/README.md @@ -138,25 +138,44 @@ dotbot run controller -w # will open a webpage at http://localhost:8000/PyDotBo Full walkthrough of fleet operations - status, OTA flash, start/stop, monitor - is in the [`swarm` reference][swarm-doc]. -## Going further +### Calibrate positions (optional) + +Give the bots real-world `(x, y)` with Lighthouse 2 - calibrate one bot over the +air, then push it to the fleet (needs the `[calibrate]` extra, below): -- **Download the firmware repository**: ```bash -git clone --recurse-submodules --branch develop https://github.com/DotBots/DotBot-firmware.git +dotbot swarm stop # the bot must be idle to capture +dotbot swarm lh2-calibration collect --device -d 500 --push ``` -- Install [SEGGER Embedded Studio](https://www.segger.com/products/development-tools/embedded-studio/), for commands such as `dotbot fw build` -- **Drive a single bot** - build, flash, and control one DotBot end to end: + +`-d` is your reference square's side, in mm. Full walkthrough - arena sizing and +the cabled alternative - is in the [LH2 calibration guide][lh2-doc]. + +## Going further + +- **Drive a single bot** end to end - build, flash, and control one DotBot: the [one-bot guide][one-bot-doc]. -- **Lighthouse 2 localization** - give your bots real-world `(x, y)` positions: - the [LH2 calibration guide][lh2-doc]. -- **Everything else** - the `dotbot` commands (`fw` / `device` / `swarm` / `run`, - plus `config`), the controller + web UI, and hardware notes are in the +- **Position tracking with Lighthouse 2** - give the fleet real-world `(x, y)`, + calibrated over the air: the [LH2 calibration guide][lh2-doc] (a cabled + alternative is covered there too). +- **The controller + web UI** - drive and visualize a swarm from the browser: + the [controller guide][controller-doc]. +- **Build firmware from source** instead of `dotbot fw fetch` - needs + [SEGGER Embedded Studio](https://www.segger.com/products/development-tools/embedded-studio/) + and a DotBot-firmware checkout: + ```bash + git clone --recurse-submodules --branch develop https://github.com/DotBots/DotBot-firmware.git + export DOTBOT_FIRMWARE_REPO=$(pwd)/DotBot-firmware + ``` + then `dotbot fw build` / `dotbot fw artifacts` (see [`fw`][fw-doc]). +- **Everything else** - the full `dotbot` CLI (`fw` / `device` / `swarm` / `run` + + `config`), the REST/WS and MQTT surfaces, and hardware notes: the [documentation][doc-link]. -Swarm orchestration is in the base install. Only LH2 calibration needs an extra: +Most of `dotbot` is in the base install; only LH2 calibration needs an extra: ```bash -pip install --pre 'pydotbot[calibrate]' # opencv-python + textual (LH2 calibration) +pip install --pre 'pydotbot[calibrate]' # opencv (the LH2 homography solve) ``` Hitting a snag (e.g. the web UI not loading in Firefox)? See @@ -184,10 +203,8 @@ See `LICENSE` in each component repository. [license-link]: https://github.com/DotBots/pydotbot/blob/main/LICENSE.txt [codecov-badge]: https://codecov.io/gh/DotBots/PyDotBot/branch/main/graph/badge.svg [codecov-link]: https://codecov.io/gh/DotBots/PyDotBot -[dotbot-firmware-repo]: https://github.com/DotBots/DotBot-firmware [cli-doc]: https://pydotbot.readthedocs.io/en/latest/cli/index.html [fw-doc]: https://pydotbot.readthedocs.io/en/latest/cli/fw.html -[device-doc]: https://pydotbot.readthedocs.io/en/latest/cli/device.html [swarm-doc]: https://pydotbot.readthedocs.io/en/latest/cli/swarm.html [config-doc]: https://pydotbot.readthedocs.io/en/latest/reference/configuration.html [simulator-doc]: https://pydotbot.readthedocs.io/en/latest/guides/simulator.html @@ -195,5 +212,3 @@ See `LICENSE` in each component repository. [one-bot-doc]: https://pydotbot.readthedocs.io/en/latest/guides/one-bot.html [lh2-doc]: https://pydotbot.readthedocs.io/en/latest/guides/lh2-calibration.html [troubleshooting-doc]: https://pydotbot.readthedocs.io/en/latest/reference/troubleshooting.html -[rest-doc]: https://pydotbot.readthedocs.io/en/latest/reference/rest.html -[mqtt-doc]: https://pydotbot.readthedocs.io/en/latest/reference/mqtt.html diff --git a/doc/cli/fw.md b/doc/cli/fw.md index e3eeca2c..d43fb314 100644 --- a/doc/cli/fw.md +++ b/doc/cli/fw.md @@ -176,4 +176,4 @@ this machine. - [`dotbot device`](device.md) - flash an artifact onto one cabled board. - [`dotbot swarm`](swarm.md) - push a sandbox app to the fleet over the air. -- [LH2 calibration](../guides/lh2-calibration.md) - the `lh2_calibration` app workflow. +- [LH2 calibration (cabled)](../guides/lh2-calibration-cabled.md) - the `lh2_calibration` app workflow. diff --git a/doc/cli/run.md b/doc/cli/run.md index 8959dcc6..02db38cd 100644 --- a/doc/cli/run.md +++ b/doc/cli/run.md @@ -14,7 +14,7 @@ dotbot run --help # the full list | `controller` | Control plane: REST/WS API + web dashboard. The hub everything else talks to. | | `gateway` | Host bridge: gateway firmware UART ↔ MQTT broker. | | `simulator` | Standalone simulator (no hardware). | -| `lh2-calibration` | Lighthouse calibration: capture / apply, on one cabled board. | +| `lh2-calibration` | LH2 calibration on one cabled board (capture / apply); deployed bots use `swarm lh2-calibration`. | | `demo` | Built-in research demos (qrkey phone bridge, …). | | `keyboard` | Drive a DotBot from the keyboard. | | `joystick` | Drive a DotBot from a joystick. | @@ -63,19 +63,21 @@ so it shares the controller's flags and serves the same dashboard. dotbot run simulator -w ``` -## `lh2-calibration` - capture & apply +## `lh2-calibration` - capture & apply (cabled) Lighthouse v2 calibration against a single serial-attached board. `collect` opens a TUI to capture LH2 counts; `apply` writes the saved calibration out as -a C header. +a C header. This is the cabled, bench path - for deployed bots, capture over the +air with [`swarm lh2-calibration`](swarm.md) instead. ```bash dotbot run lh2-calibration collect dotbot run lh2-calibration apply ./lh2_calibration.h ``` -See [the LH2 calibration guide](../guides/lh2-calibration.md). To push a -calibration to the whole fleet over the air, use [`swarm calibrate-lh2`](swarm.md). +See [the cabled LH2 calibration guide](../guides/lh2-calibration-cabled.md). To +capture without a cable, or to push a saved calibration to the fleet over the +air, use [`swarm lh2-calibration`](swarm.md). ## `demo` - built-in demos diff --git a/doc/cli/swarm.md b/doc/cli/swarm.md index fb42e3b3..badaa5e4 100644 --- a/doc/cli/swarm.md +++ b/doc/cli/swarm.md @@ -101,18 +101,24 @@ To replace a running experiment: `stop`, then `flash ... -ys`. | `-t`, `--ota-timeout` | seconds per OTA ACK (default `0.7`) | | `-r`, `--ota-max-retries` | retries per OTA message (default `10`) | -## 6. Push an LH2 calibration over the air +## 6. LH2 calibration over the air -Send a calibration (captured from one cabled bot - see -[LH2 calibration](../guides/lh2-calibration.md)) to the whole fleet: +Capture and push a Lighthouse-2 calibration for one DotBot without a cable, +driving it over the swarm. The arena geometry and `-d` sizing live in the +[LH2 calibration guide](../guides/lh2-calibration.md). ```bash -dotbot swarm stop -dotbot swarm calibrate-lh2 ~/.dotbot/calibration-.toml +dotbot swarm stop # capture only runs in READY +dotbot swarm lh2-calibration collect --device BC3D... -d 500 # capture from one bot -> solve -> save +dotbot swarm lh2-calibration push ~/.dotbot/calibration-.toml # apply to every ready bot ``` -It accepts a `calibration-*.toml` or the legacy raw payload; the format is -picked by file extension. +`collect` walks one bot through the four arena corners over the air, solves the +homography, and saves it under `~/.dotbot/`. `push` (no `--device`) then sends +that calibration to **every ready bot** - the arena shares one transform. +(`collect --push` is a single-bot shortcut: it sends only to the captured bot.) +`push` takes a `calibration-*.toml` or the legacy raw payload - the format is +picked by file extension. Get the `--device` address from `dotbot swarm status`. ## Two web servers - don't mix them up diff --git a/doc/guides/index.md b/doc/guides/index.md index cf8085d4..60fe7321 100644 --- a/doc/guides/index.md +++ b/doc/guides/index.md @@ -8,6 +8,7 @@ simulator one-bot controller lh2-calibration +lh2-calibration-cabled ``` - [Try it in the simulator](simulator.md) - run the full UI and script bots with @@ -17,4 +18,6 @@ lh2-calibration - [Run the controller + web UI](controller.md) - drive and visualize a swarm from the browser. - [Lighthouse 2 localization](lh2-calibration.md) - give your bots real-world - `(x, y)` positions. + `(x, y)` positions, calibrated over the air. +- [LH2 calibration over a cable](lh2-calibration-cabled.md) - the bench + alternative, for a single USB-connected bot. diff --git a/doc/guides/lh2-calibration-cabled.md b/doc/guides/lh2-calibration-cabled.md new file mode 100644 index 00000000..9f20122c --- /dev/null +++ b/doc/guides/lh2-calibration-cabled.md @@ -0,0 +1,87 @@ +# LH2 calibration over a serial cable + +The bench alternative to the [over-the-air flow](lh2-calibration.md): calibrate a +single DotBot connected over USB, with no swarm provisioned. Reach for this when +you're working with one bot on the bench, or before the fleet is set up. For +already-deployed bots, prefer the over-the-air flow - no cable, no firmware swap. + +What LH2 calibration is, and the arena geometry (the `-d` square sizing), are +covered in the [main guide](lh2-calibration.md); this page is just the cabled +capture path. + +## Prerequisites + +- A DotBot v3 you can cable to your machine over USB-C (no external probe - the + v3 flashes over its on-board programmer). +- Two LH2 base stations facing the arena, and a square marked on the floor (see + [Sizing `-d`](lh2-calibration.md) in the main guide). +- The `[calibrate]` extra: + +```bash +pip install --pre 'pydotbot[calibrate]' +``` + +## 1. Flash the capture firmware + +The `lh2_calibration` app streams raw LH2 counts over serial. Flash it to the +cabled bot (see [device](../cli/device.md) for serial-prefix selection): + +```bash +dotbot device flash lh2_calibration -s 77 # board defaults to dotbot-v3 +``` + +## 2. Capture the four reference points + +Place the bot on the floor square and run the TUI. `-d` is the side length of +the square, in millimeters: + +```bash +dotbot run lh2-calibration collect -p /dev/cu.usbmodem... -d 500 +``` + +Move the bot to each corner - Top left -> Top right -> Bottom left -> Bottom +right - pressing the matching button in the TUI at each. When all four are +captured, save. The calibration is written under `~/.dotbot/` (a +`calibration-.toml`), the same place the over-the-air flow uses. + +| Flag | Default | Meaning | +|---|---|---| +| `-p`, `--port` | auto-detect | Serial port of the calibration firmware. | +| `-d`, `--distance` | calibration default | Square side length, **in mm**. | +| `-n`, `--extra-lh-num` | `0` | Extra base stations beyond the first (0–5). | +| `--input-data` | - | Re-process a saved capture instead of capturing live. | + +See `dotbot run lh2-calibration collect --help` for the full list. + +## 3. Use the calibration + +Send the saved calibration to the fleet over the air (the same command the +over-the-air flow uses - stop any running app first): + +```bash +dotbot swarm stop +dotbot swarm lh2-calibration push ~/.dotbot/calibration-.toml +``` + +### Bake it into the bootloader (header path) + +For a fresh board whose bootloader bakes the calibration in at compile time +(rather than receiving it over the air), export the saved calibration as a C +header instead: + +```bash +dotbot run lh2-calibration apply ./lh2_calibration.h +``` + +The swarmit secure bootloader `#include`s this file; rebuild and reflash the +bootloader for it to take effect. For already-running bots, prefer the +over-the-air push above - no reflash needed. + +## Troubleshooting + +- **No counts in the TUI** - wrong `-p` port, or the bot can't see both base + stations. Confirm line-of-sight and that the base-station LEDs are steady. +- **Positions look skewed or mirrored** - the corners were captured out of + order. Re-run `collect` and follow TL -> TR -> BL -> BR exactly. +- **Positions are scaled wrong** - `-d` didn't match the real square. It's in + millimeters, not centimeters. diff --git a/doc/guides/lh2-calibration.md b/doc/guides/lh2-calibration.md index 647b7083..f39eb36d 100644 --- a/doc/guides/lh2-calibration.md +++ b/doc/guides/lh2-calibration.md @@ -3,58 +3,82 @@ Lighthouse 2 gives every DotBot a real-world **(x, y) position** on your arena floor. Two SteamVR base stations sweep the room with IR; each bot's LH2 sensor times the sweeps. Calibration is the one-time step that maps those raw sweep -counts to metric coordinates: you place one bot on four known points of a +counts to metric coordinates: you place one bot on four known corners of a square, capture, and the resulting transform is pushed to the whole fleet. -You do this once per physical setup (move a base station → recalibrate). +You do this once per physical setup (move a base station -> recalibrate). + +**The default flow is over the air** - drive one already-deployed bot through the +corners over the swarm, no cable and no firmware swap. If you'd rather calibrate +a single bot on the bench over USB, see +[LH2 calibration over a cable](lh2-calibration-cabled.md). ## Prerequisites -- A DotBot v3 you can cable to your machine over USB-C (no external probe - needed - the v3 flashes over its on-board programmer). +- A provisioned swarm: a gateway plus sandbox-host bots, reachable from your + config (see [swarm](../cli/swarm.md)). - Two LH2 base stations mounted ~2 m up, facing the arena. -- A square marked on the floor with a known side length, plus the - `[calibrate]` extra installed: +- A square marked on the floor with a known side length. +- The `[calibrate]` extra (the homography solve uses opencv): ```bash pip install --pre 'pydotbot[calibrate]' ``` -## 1. Flash the capture firmware +## Capture and push -The `lh2_calibration` app streams raw LH2 counts over serial. Flash it to the -cabled bot (see [device](../cli/device.md) for serial-prefix selection): +The calibration is a property of the **arena** (the base-station layout), not the +individual bot, so you capture once from any one bot and apply the result to the +whole fleet. Two steps: ```bash -dotbot device flash lh2_calibration -s 77 # board defaults to dotbot-v3 +dotbot swarm stop # put the bots in READY +dotbot swarm lh2-calibration collect \ + --device BC3D3C8A2A6F8E68 -d 500 # capture from one bot -> solve -> save +dotbot swarm lh2-calibration push \ + ~/.dotbot/calibration-.toml # apply to every ready bot ``` -## 2. Capture the four reference points - -Place the bot on the floor square and run the TUI. `-d` is the **side length of -the square, in millimeters**: - -```bash -dotbot run lh2-calibration collect -p /dev/cu.usbmodem... -d 500 +`collect` walks one bot through the four corners - **top-left -> top-right -> +bottom-left -> bottom-right** - (capture only runs while the bot is in READY, so +`swarm stop` first). Each prompt triggers a raw-count capture over the air; it +then solves the homography and saves a `calibration-.toml` under `~/.dotbot/` +(the path is printed at the end). Find the `--device` address with +`dotbot swarm status`. + +`push` with **no `--device`** sends that calibration to **every ready bot** - the +whole arena shares one transform. It accepts a `calibration-*.toml` or the legacy +raw `calibration.out` payload; the format is picked by file extension. + +```{note} +`collect --push` is a **single-bot shortcut**: it sends the result to *only* the +`--device` bot you captured from (handy to spot-check that one bot, or for a +single-bot setup). To calibrate the fleet, run the standalone `push` above - it +targets all ready bots. ``` -Move the bot to each corner - Top left → Top right → Bottom left → Bottom right -- pressing the matching button in the TUI at each. When all four are captured, -save. The calibration is written under `~/.dotbot/` (a `calibration-.toml`). +Once pushed, the bots report positions, which show up live in the +[controller](../cli/run.md) Web UI. -Common `collect` flags: +### `collect` flags | Flag | Default | Meaning | |---|---|---| -| `-p`, `--port` | auto-detect | Serial port of the calibration firmware. | -| `-d`, `--distance` | - | Square side length, **in mm** (see sizing below). | -| `-n`, `--extra-lh-num` | `0` | Extra base stations beyond the first (0–5). | -| `--input-data` | - | Re-process a saved capture instead of capturing live. | +| `--device` | (required) | DotBot link-layer address in hex (from `dotbot swarm status`). | +| `-d`, `--distance` | calibration default | Square side length, **in mm** (see sizing below). | +| `-n`, `--conn` / `-s`, `--swarm-id` | from config | Swarm connection, like the other `dotbot swarm` commands. | +| `--timeout` | `5` s | Seconds to wait for each capture before re-triggering. | +| `--retries` | `3` | Re-trigger this many times per corner before giving up. | +| `--tag` | - | Arena/setup label (e.g. `office-2x2m`) added to the filename + metadata. | +| `--push` | off | After solving, send to the captured `--device` bot **only** (use the standalone `push` for the whole fleet). | -See `dotbot run lh2-calibration collect --help` for the full list. +See `dotbot swarm lh2-calibration collect --help` for the full list. -**Sizing `-d`** - the usable arena is **5× the square side**, with the square -centered (a `2·d` margin on every side): +## Sizing `-d` + +`-d` is the **side of your reference square, in millimeters**. The usable arena +is **5× the square side**, with the square centered (a `2·d` margin on every +side): ``` ←─────────────── 5·d ────────────→ @@ -82,42 +106,12 @@ square side (`--distance`, in mm), `5·d` the resulting arena. | `500` | 50 cm | 2.5 m × 2.5 m | | `800` | 80 cm | 4.0 m × 4.0 m (used for the 725-bot Limerick run) | -## 3. Push the calibration to the fleet - -Send the captured calibration over the air. Stop any running app first, then -push the `.toml` (see [swarm](../cli/swarm.md) for the connection config): - -```bash -dotbot swarm stop -dotbot swarm calibrate-lh2 ~/.dotbot/calibration-.toml -``` - -`calibrate-lh2` accepts either a `calibration-*.toml` or the legacy raw -`calibration.out` payload - the format is picked by file extension. - -Once pushed, the bots report positions, which show up live in the -[controller](../cli/run.md) Web UI. - -## First-time flashing (header path) - -For a fresh board whose bootloader bakes the calibration in at compile time -(rather than receiving it over the air), export the saved calibration as a C -header instead: - -```bash -dotbot run lh2-calibration apply ./lh2_calibration.h -``` - -The swarmit secure bootloader `#include`s this file; rebuild and reflash the -bootloader for it to take effect. For already-running bots, prefer the OTA path -in step 3 - no reflash needed. - ## Troubleshooting -- **No counts in the TUI** - wrong `-p` port, or the bot can't see both base - stations. Confirm line-of-sight and that the LEDs on the base stations are - steady. +- **Capture times out** - the bot isn't in READY (run `dotbot swarm stop` + first), or it can't see both base stations. The address passed to `--device` + must match one from `dotbot swarm status`. - **Positions look skewed or mirrored** - the corners were captured out of - order. Re-run `collect` and follow TL → TR → BL → BR exactly. + order. Re-run `collect` and follow TL -> TR -> BL -> BR exactly. - **Positions are scaled wrong** - `-d` didn't match the real square. It's in millimeters, not centimeters. diff --git a/doc/guides/one-bot.md b/doc/guides/one-bot.md index 5d56c066..7b059608 100644 --- a/doc/guides/one-bot.md +++ b/doc/guides/one-bot.md @@ -2,7 +2,7 @@ Build and flash one DotBot and a gateway, cable them to your computer, and drive the bot from the web UI. This is the smallest real-hardware setup. For the -no-hardware path, use the [simulator](controller.md) instead. +no-hardware path, use the [simulator](simulator.md) instead. Building firmware needs SEGGER Embedded Studio and `nrfjprog` (see the README prerequisites). To skip building, fetch a pre-built release with `dotbot fw @@ -49,4 +49,5 @@ Select the bot in the browser and steer it with the joystick. See the ## Next - Operate many bots over the air - the [`swarm`](../cli/swarm.md) reference. -- Add real-world positions - [LH2 calibration](lh2-calibration.md). +- Add real-world positions - [LH2 calibration over a cable](lh2-calibration-cabled.md) + (or [over the air](lh2-calibration.md) once you've provisioned a swarm). From f93abc872a5d97e4265ac035f6da83d84b571c22 Mon Sep 17 00:00:00 2001 From: Geovane Fedrecheski Date: Thu, 4 Jun 2026 12:53:56 +0200 Subject: [PATCH 27/27] readme: add over-the-air LH2 calibration + rework Going further AI-assisted: Claude Opus 4.8 --- README.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 27364e0c..35f7558b 100644 --- a/README.md +++ b/README.md @@ -140,16 +140,19 @@ is in the [`swarm` reference][swarm-doc]. ### Calibrate positions (optional) -Give the bots real-world `(x, y)` with Lighthouse 2 - calibrate one bot over the -air, then push it to the fleet (needs the `[calibrate]` extra, below): +Give the bots real-world `(x, y)` with Lighthouse 2 - capture once from any bot +over the air, then push the result to the whole fleet (needs the `[calibrate]` +extra, below): ```bash -dotbot swarm stop # the bot must be idle to capture -dotbot swarm lh2-calibration collect --device -d 500 --push +dotbot swarm stop # bots must be idle to capture +dotbot swarm lh2-calibration collect --device -d 500 # capture + solve + save +dotbot swarm lh2-calibration push ~/.dotbot/calibration-.toml # apply to every bot ``` -`-d` is your reference square's side, in mm. Full walkthrough - arena sizing and -the cabled alternative - is in the [LH2 calibration guide][lh2-doc]. +`-d` is your reference square's side, in mm (one bot's capture calibrates the +whole arena). Full walkthrough - arena sizing and the cabled alternative - is in +the [LH2 calibration guide][lh2-doc]. ## Going further