diff --git a/README.md b/README.md index 20b1f2e0..35f7558b 100644 --- a/README.md +++ b/README.md @@ -6,19 +6,13 @@ # PyDotBot -**The control plane for [DotBot](http://www.dotbots.org) swarms - build firmware, -flash a robot, and control a fleet over the air, from one bot to a thousand, all -from a single `dotbot` CLI and web UI.** +The control plane for the [DotBot](http://www.dotbots.org) - a small wireless +wheeled robot built to operate in large swarms, for research and education. -DotBots are small wireless wheeled robots built to operate in large swarms, -for research and education. Developed by the [AIO team](https://aio.inria.fr/) at -[Inria Paris](https://www.inria.fr/), and run routinely with ~100–200 bots, -with one 725-bot campaign. +PyDotBot allows you to flash a robot and control a whole fleet over the air, +from one bot to a thousand. -▶️ [Click to see a DotBot swarm in action](https://www.youtube.com/watch?v=pXGTLqafReU) - -PyDotBot is the control plane in the middle: your code, the web UI, and users -talk to it, and it drives the swarm through a gateway. +[▶️ Click to see a DotBot swarm in action](https://www.youtube.com/watch?v=pXGTLqafReU) ```text ┌───────────┐ ┌────────────┐ ┌─────────┐ @@ -37,125 +31,53 @@ talk to it, and it drives the swarm through a gateway. - 🧪 Try it all with **zero hardware** using the built-in simulator - 🛠️ One `dotbot` CLI takes you from build → flash → run -## Try it now - no hardware +## Install -See the whole thing run with nothing but Python: +PyDotBot is available on [PyPi](https://pypi.org/project/pydotbot/), install it with: ```bash -pip install --pre pydotbot # using 'pre' while we are at release candidate -dotbot run simulator -w # opens the web UI at http://localhost:8000/PyDotBot/, driving a simulated swarm -``` - -Drive the simulated bots from the joystick + map - then script them from your own -code (below), or set up real hardware further down. - -## Drive it from your own code - -The controller - real or simulated - exposes a REST + WebSocket API, so you can -command the swarm in a few lines of Python (only extra dependency: -[`requests`](https://pypi.org/project/requests/)): - -```python -import requests, time - -BASE = "http://localhost:8000" -bot = requests.get(f"{BASE}/controller/dotbots").json()[0]["address"] - -# roll in a circle for ~5 s - left_y and right_y are the two wheel speeds -for _ in range(50): - requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", - json={"left_x": 0, "left_y": 60, "right_x": 0, "right_y": 30}) - time.sleep(0.1) -requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", - json={"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}) +pip install --pre pydotbot ``` -The full surface - every endpoint, the live WebSocket stream, and CSV data -logging - is in the [REST / WebSocket reference][rest-doc] (or the -[MQTT bridge][mqtt-doc]). A higher-level Python SDK is planned; today you talk to -the controller over REST/WebSocket/MQTT. - -The firmware for the DotBots can be found [here][dotbot-firmware-repo]. +Then, check your installation with `dotbot --version` and learn what's possible with `dotbot --help`. -## Prerequisites (for real hardware) - -Driving an already-provisioned swarm - or the simulator above - needs nothing but -Python. The tools below are only for building or cable-flashing firmware yourself. +Every command and flag is documented in the [CLI reference][cli-doc]. -Software to install (as needed): -- Python ≥ 3.11 - ensure you also have [pip](https://pip.pypa.io/en/stable/) available in your PATH -- [nRF Command Line Tools](https://www.nordicsemi.com/Products/Development-tools/nRF-Command-Line-Tools) (`nrfjprog`), for commands such as `dotbot device flash` -- [SEGGER Embedded Studio](https://www.segger.com/products/development-tools/embedded-studio/), for commands such as `dotbot fw build` +## Try the simulator -Minimal hardware setup: -- DotBot v3, as well as a USB-C cable and a barrel-jack charger (2.5 mm, 6–18 V, 5/10 A) -- nRF5340-DK to use as gateway, as well as a micro-USB cable +See the whole thing run with nothing but Python! -## Install +The command below will run a simulated swarm, which you can observe in a web UI at http://localhost:8000/PyDotBot/ : ```bash -pip install --pre pydotbot # --pre while 0.29 is in pre-release -git clone --recurse-submodules --branch develop https://github.com/DotBots/DotBot-firmware.git +dotbot run simulator -w ``` -## Usage - -``` -$ dotbot --help -Usage: dotbot [OPTIONS] COMMAND [ARGS]... - - One CLI for the whole DotBot workflow: build and flash firmware, program and - control a single robot, and run experiments over the air across a swarm - - from one bot to a thousand. - -Commands: - fw Firmware artifacts (no hardware): build / fetch / list / make. - device One connected device (cable/probe): flash an app/role, read info. - swarm The fleet over the air: status, start/stop, OTA flash, monitor. - run Host-side processes: controller, gateway, simulator, calibration, demos, teleop. - config Show the resolved config and where it came from; scaffold one with init. -``` - -Every command and flag is documented in the [CLI reference][cli-doc]. - -## Quickstart - one bot - -Build and flash firmware for a single dotbot: +Drive the simulated bots from the UI, or run a bundled demo in a +second terminal: ```bash -# build the bare dotbot apps into ./artifacts/ (needs SEGGER Embedded Studio) -# two steps because the DotBot has two cores -dotbot fw artifacts --app dotbot -dotbot fw artifacts --app nrf5340_net --target nrf5340dk-net -# cable-flash it to the bot whose J-Link serial starts with 77 -dotbot device flash dotbot -s 77 # app core -dotbot device flash nrf5340_net -b nrf5340dk-net -s 77 # network core +dotbot run demo circle # drive one bot in a circle (the simplest demo) ``` -Now, build and flash the gateway to connect to a robot. -The gateway is a dev board (e.g. an nRF52840-DK) plugged into your -computer; it bridges the robot's radio to USB serial. +Learn how to script the swarm from your own code, run the richer examples, and more - all with +no hardware - in the [simulator guide][simulator-doc]. -```bash -# build the gateway firmware for your DK board into ./artifacts/ (needs SEGGER Embedded Studio) -dotbot fw artifacts --app dotbot_gateway --target nrf52840dk -# cable-flash it to the DK whose J-Link serial starts with 10 -dotbot device flash dotbot_gateway -b nrf52840dk -s 10 -``` +## Deploy a real swarm -With a gateway plugged into your computer, point the controller at it -and open the web UI: +The DotBot is made to operate as a swarm, here is how you can deploy it on real robots. -```bash -dotbot run controller --conn /dev/ttyACM0 -w # serial gateway; no swarm-id needed -``` +### Prerequisites -More detail: building and flashing one board ([`fw`][fw-doc] / [`device`][device-doc]) -and driving it from the web UI ([controller guide][controller-doc]). +Minimal hardware setup: +- DotBot v3, as well as a USB-C cable and a barrel-jack charger (2.5 mm, 6–18 V, 5/10 A) +- nRF5340-DK to use as gateway, as well as a micro-USB cable -## Quickstart - a swarm +Software to install (as needed): +- Python ≥ 3.11 - ensure you also have [pip](https://pip.pypa.io/en/stable/) available in your PATH +- [nRF Command Line Tools](https://www.nordicsemi.com/Products/Development-tools/nRF-Command-Line-Tools) (`nrfjprog`), for commands such as `dotbot device flash` -### setup the swarm +### Setup To operate as a swarm, set your swarm connection config: @@ -173,13 +95,13 @@ We also need a more powerful gateway firmware. Let's flash both - the network id comes from your config: ```bash -dotbot fw fetch -f 0.8.0rc1 # pull the pre-compiled firmwares from a release -dotbot device flash-mari-gateway -s 10 -f 0.8.0rc1 # flash the gateway -dotbot device flash-swarmit-sandbox -s 77 -f 0.8.0rc1 # the sandbox firmware - do this on each dotbot +dotbot fw fetch # pull the pinned pre-compiled firmwares (swarmit + dotbot-firmware) +dotbot device flash-mari-gateway -s 10 # flash the gateway +dotbot device flash-swarmit-sandbox -s 77 # the sandbox firmware - do this on each dotbot ``` (`device flash-mari-gateway` / `flash-swarmit-sandbox` auto-fetch -the release into `./artifacts/` if it isn't already there.) +the firmware into `~/.dotbot/artifacts/` if it isn't already there.) Now, run the gateway (the broker comes from your config): @@ -187,22 +109,24 @@ Now, run the gateway (the broker comes from your config): dotbot run gateway -p /dev/cu.usbmodem0010500324491 ``` -### use the swarm +### Deploy and control You can flash as many dotbots as you want, all at once! First, how about making them spinnnn 🔄 🔄 ```bash -dotbot swarm flash ./artifacts/spin-sandbox-dotbot-v3.bin -ys # flash the whole fleet with a simple spinning app +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-1.22.0rc1/spin-sandbox-dotbot-v3.bin -ys # flash the whole fleet with a simple spinning app ``` (`dotbot swarm` reads the same `dotbot.toml` as the rest - pass `--conn` / -`--swarm-id` to override it for one run.) +`--swarm-id` to override it for one run. That path is the `dotbot-firmware` +release `dotbot fw fetch` cached - run `dotbot fw list` to see the exact paths +and versions on your machine.) Then, flash another experiment: ```bash dotbot swarm stop # ensure all robots are in bootloader -dotbot swarm flash ./artifacts/dotbot-sandbox-dotbot-v3.bin -ys # this firmware allows bots to be remote-controlled +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-1.22.0rc1/dotbot-sandbox-dotbot-v3.bin -ys # this firmware allows bots to be remote-controlled ``` Observe and control your swarm from a web interface: @@ -214,36 +138,47 @@ dotbot run controller -w # will open a webpage at http://localhost:8000/PyDotBo Full walkthrough of fleet operations - status, OTA flash, start/stop, monitor - is in the [`swarm` reference][swarm-doc]. -## Quickstart - Lighthouse 2 localization +### Calibrate positions (optional) -Give your robots a real-world `(x, y)` position. You'll need at least one -Lighthouse 2 base station and the calibration extra -(`pip install --pre 'pydotbot[calibrate]'`). +Give the bots real-world `(x, y)` with Lighthouse 2 - capture once from any bot +over the air, then push the result to the whole fleet (needs the `[calibrate]` +extra, below): ```bash -# 1. flash the capture firmware to a cabled dotbot and collect four corner points -dotbot device flash lh2_calibration -s 77 -dotbot run lh2-calibration collect -p /dev/tty.usbmodem0007745943981 -d 200 # square of side 20 cm - -# 2. push the resulting calibration to the fleet over the air -dotbot swarm stop # ensure all robots are in bootloader -dotbot swarm calibrate-lh2 ~/.dotbot/calibration-2026-05-26T14-00-36Z.toml +dotbot swarm stop # bots must be idle to capture +dotbot swarm lh2-calibration collect --device -d 500 # capture + solve + save +dotbot swarm lh2-calibration push ~/.dotbot/calibration-.toml # apply to every bot ``` -Your bots now report their `(x, y)` location. The full setup - arena sizing, -base-station placement, and troubleshooting - is in the -[LH2 calibration guide][lh2-doc]. +`-d` is your reference square's side, in mm (one bot's capture calibrates the +whole arena). Full walkthrough - arena sizing and the cabled alternative - is in +the [LH2 calibration guide][lh2-doc]. ## Going further -Full command reference and guides - running the controller + web UI, the four -CLI namespaces (`fw` / `device` / `swarm` / `run`), hardware, and LH2 -calibration - are in the [documentation][doc-link]. - -Swarm orchestration is in the base install. Only LH2 calibration needs an extra: +- **Drive a single bot** end to end - build, flash, and control one DotBot: + the [one-bot guide][one-bot-doc]. +- **Position tracking with Lighthouse 2** - give the fleet real-world `(x, y)`, + calibrated over the air: the [LH2 calibration guide][lh2-doc] (a cabled + alternative is covered there too). +- **The controller + web UI** - drive and visualize a swarm from the browser: + the [controller guide][controller-doc]. +- **Build firmware from source** instead of `dotbot fw fetch` - needs + [SEGGER Embedded Studio](https://www.segger.com/products/development-tools/embedded-studio/) + and a DotBot-firmware checkout: + ```bash + git clone --recurse-submodules --branch develop https://github.com/DotBots/DotBot-firmware.git + export DOTBOT_FIRMWARE_REPO=$(pwd)/DotBot-firmware + ``` + then `dotbot fw build` / `dotbot fw artifacts` (see [`fw`][fw-doc]). +- **Everything else** - the full `dotbot` CLI (`fw` / `device` / `swarm` / `run` + + `config`), the REST/WS and MQTT surfaces, and hardware notes: the + [documentation][doc-link]. + +Most of `dotbot` is in the base install; only LH2 calibration needs an extra: ```bash -pip install --pre 'pydotbot[calibrate]' # opencv-python + textual (LH2 calibration) +pip install --pre 'pydotbot[calibrate]' # opencv (the LH2 homography solve) ``` Hitting a snag (e.g. the web UI not loading in Firefox)? See @@ -271,14 +206,12 @@ See `LICENSE` in each component repository. [license-link]: https://github.com/DotBots/pydotbot/blob/main/LICENSE.txt [codecov-badge]: https://codecov.io/gh/DotBots/PyDotBot/branch/main/graph/badge.svg [codecov-link]: https://codecov.io/gh/DotBots/PyDotBot -[dotbot-firmware-repo]: https://github.com/DotBots/DotBot-firmware [cli-doc]: https://pydotbot.readthedocs.io/en/latest/cli/index.html [fw-doc]: https://pydotbot.readthedocs.io/en/latest/cli/fw.html -[device-doc]: https://pydotbot.readthedocs.io/en/latest/cli/device.html [swarm-doc]: https://pydotbot.readthedocs.io/en/latest/cli/swarm.html [config-doc]: https://pydotbot.readthedocs.io/en/latest/reference/configuration.html +[simulator-doc]: https://pydotbot.readthedocs.io/en/latest/guides/simulator.html [controller-doc]: https://pydotbot.readthedocs.io/en/latest/guides/controller.html +[one-bot-doc]: https://pydotbot.readthedocs.io/en/latest/guides/one-bot.html [lh2-doc]: https://pydotbot.readthedocs.io/en/latest/guides/lh2-calibration.html [troubleshooting-doc]: https://pydotbot.readthedocs.io/en/latest/reference/troubleshooting.html -[rest-doc]: https://pydotbot.readthedocs.io/en/latest/reference/rest.html -[mqtt-doc]: https://pydotbot.readthedocs.io/en/latest/reference/mqtt.html diff --git a/doc/cli/device.md b/doc/cli/device.md index 5a06bef1..ca1f94ab 100644 --- a/doc/cli/device.md +++ b/doc/cli/device.md @@ -32,8 +32,9 @@ UART↔MQTT bridge process is a different thing - that's [`run gateway`](run.md) ## Flash an app -`flash` resolves `` to `./artifacts/-.hex`, **building it if the -file isn't there**; an explicit `.hex`/`.bin` path is flashed as-is. +`flash` resolves `` to a cached `-.hex` (under +`~/.dotbot/artifacts/`), **building it if the file isn't there**; an explicit +`.hex`/`.bin` path is flashed as-is. ```bash export DOTBOT_FIRMWARE_REPO=$(pwd)/repos/DotBot-firmware @@ -86,7 +87,7 @@ dotbot device flash nrf5340_net -b nrf5340dk-net -s 10 `flash-mari-gateway` and `flash-swarmit-sandbox` flash a **complete system firmware** (both cores) and write the **network identity** in one shot. They auto-fetch the -named release into `./artifacts/` if it isn't cached. +named release into `~/.dotbot/artifacts/` if it isn't cached. ```bash # nRF5340-DK → swarm gateway @@ -99,7 +100,7 @@ dotbot device flash-swarmit-sandbox --swarm-id 0100 -f 0.8.0rc1 -s 77 | Flag | `flash-mari-gateway` | `flash-swarmit-sandbox` | |---|---|---| | `--swarm-id` | 16-bit hex swarm id (or from config) | 16-bit hex swarm id (or from config) | -| `-f, --fw-version` | release to flash (required) | release to flash (required) | +| `-f, --fw-version` | release to flash (default: the pinned swarmit version) | release to flash (default: the pinned swarmit version) | | `-s, --sn-starting-digits` | J-Link serial prefix | J-Link serial prefix | | `-l, --calibration` | - | optional LH2 calibration file to bake in | diff --git a/doc/cli/fw.md b/doc/cli/fw.md index 6fe4570b..d43fb314 100644 --- a/doc/cli/fw.md +++ b/doc/cli/fw.md @@ -43,8 +43,8 @@ segger_dir = "/path/to/SEGGER Embedded Studio X.YY" | Goal | Command | |---|---| | Compile an app, leave it in the SES `Output/` tree (path echoed) | `dotbot fw build` | -| Compile **and** collect a flat `-.hex` into `./artifacts/` | `dotbot fw artifacts` | -| Download a published release into `./artifacts//` | `dotbot fw fetch -f ` | +| Compile **and** collect a flat `-.hex` into the cache | `dotbot fw artifacts` | +| Download the pinned release(s) into `~/.dotbot/artifacts/-/` | `dotbot fw fetch [-S -f ]` | | List targets you can build | `dotbot fw targets [--sandbox]` | | List what's cached locally | `dotbot fw list` | | A Makefile knob the CLI doesn't model | `dotbot fw make ` | @@ -52,9 +52,10 @@ segger_dir = "/path/to/SEGGER Embedded Studio X.YY" **`build` vs `artifacts`**: both compile via SES. `build` stops once SES is done (output stays buried in the per-target `Output/` tree). `artifacts` goes one step further and copies a flat, predictably-named `-.hex` into -`./artifacts/` - which is exactly where `dotbot device flash ` and the -swarm tools look. Reach for `artifacts` when you intend to flash; `build` when -you only want to know it compiles. +the cache (`~/.dotbot/artifacts/dotbot-firmware-local/`) - which is exactly +where `dotbot device flash ` and the swarm tools look. Reach for +`artifacts` when you intend to flash; `build` when you only want to know it +compiles. ## `build` / `artifacts` flags @@ -69,8 +70,8 @@ Both share the same build options: | `--rebuild` | Force a full rebuild (default: incremental) | | `-v, --verbose` | Full SES output | -`artifacts` adds `--out ` (default `./artifacts/`) and `--print-path` -(report where the artifact would land without building). See +`artifacts` adds `--out ` (default `~/.dotbot/artifacts/dotbot-firmware-local/`) +and `--print-path` (report where the artifact would land without building). See `dotbot fw --help` for the full list. > **Flag mismatch to remember:** `fw` selects a board with `--target/-t`, but @@ -105,12 +106,40 @@ Notes: - The nRF5340 radio lives on the **net core**, so a gateway needs two images: `dotbot_gateway` on `nrf5340dk-app` **and** `nrf5340_net` on `nrf5340dk-net`. +## `fetch` - pinned firmware + +`dotbot fw fetch` downloads prebuilt firmware from two release sources - +**swarmit** (the swarm system images) and **DotBot-firmware** (the bare and +sandbox apps) - into `~/.dotbot/artifacts/-/`, each with a +`manifest.json` recording where it came from. + +With no flags it fetches the **exact versions this `dotbot` is pinned to**, so a +given `dotbot` always pulls a known-good, reproducible set: + +- **swarmit** is also a Python dependency, so its firmware version is read from + the installed `swarmit` package. +- **DotBot-firmware** is not a Python package, so the version `dotbot` is built + and tested against is declared in `dotbot` and bumped deliberately. + +Override per source when you need something else - the two version +independently, so `-f` requires `-S`: + +```bash +dotbot fw fetch # pinned versions, both sources +dotbot fw fetch -S dotbot-firmware -f latest # newest dotbot-firmware release +dotbot fw fetch -S swarmit -f 0.8.0rc2 # a specific swarmit version +``` + +The cache is user-level and shared across projects (override the location with +`$DOTBOT_ARTIFACTS_DIR`); `dotbot device flash` and the swarm tools resolve +their input from it. + ## Examples ```bash export DOTBOT_FIRMWARE_REPO=/path/to/DotBot-firmware -# Bare DotBot app for a DotBot v3 → ./artifacts/dotbot-dotbot-v3.hex +# Bare DotBot app for a DotBot v3 → ~/.dotbot/artifacts/dotbot-firmware-local/dotbot-dotbot-v3.hex dotbot fw artifacts --app dotbot # Just confirm an app compiles (no collection) @@ -123,8 +152,8 @@ dotbot fw artifacts --app nrf5340_net -t nrf5340dk-net # Sandbox (NS) "spin" app for a DotBot v3 → .bin, for OTA via swarm dotbot fw artifacts --app spin -t dotbot-v3 --sandbox -# Pull a published release instead of building → ./artifacts// -dotbot fw fetch -f v1.0.0 +# Download released firmware instead of building (see "fetch" above) +dotbot fw fetch # the versions this dotbot is pinned to (both sources) # See what's cached dotbot fw list @@ -147,4 +176,4 @@ this machine. - [`dotbot device`](device.md) - flash an artifact onto one cabled board. - [`dotbot swarm`](swarm.md) - push a sandbox app to the fleet over the air. -- [LH2 calibration](../guides/lh2-calibration.md) - the `lh2_calibration` app workflow. +- [LH2 calibration (cabled)](../guides/lh2-calibration-cabled.md) - the `lh2_calibration` app workflow. diff --git a/doc/cli/index.md b/doc/cli/index.md index 0dc389f6..c763f1ac 100644 --- a/doc/cli/index.md +++ b/doc/cli/index.md @@ -46,8 +46,8 @@ Need a process running on my computer (UI, gateway bridge, demo)? ─► run A few signposts so the namespaces don't blur together: - **`fw` never touches hardware.** It only produces or lists artifacts in - `./artifacts/`. Flashing always happens under `device` (cabled) or `swarm` - (OTA). + `~/.dotbot/artifacts/`. Flashing always happens under `device` (cabled) or + `swarm` (OTA). - **Bare vs. sandbox artifacts.** `fw` builds bare apps (`.hex`) by default; `fw artifacts --sandbox` builds TrustZone apps (`.bin`) - the payload `swarm` flashes over the air. diff --git a/doc/cli/run.md b/doc/cli/run.md index 8959dcc6..02db38cd 100644 --- a/doc/cli/run.md +++ b/doc/cli/run.md @@ -14,7 +14,7 @@ dotbot run --help # the full list | `controller` | Control plane: REST/WS API + web dashboard. The hub everything else talks to. | | `gateway` | Host bridge: gateway firmware UART ↔ MQTT broker. | | `simulator` | Standalone simulator (no hardware). | -| `lh2-calibration` | Lighthouse calibration: capture / apply, on one cabled board. | +| `lh2-calibration` | LH2 calibration on one cabled board (capture / apply); deployed bots use `swarm lh2-calibration`. | | `demo` | Built-in research demos (qrkey phone bridge, …). | | `keyboard` | Drive a DotBot from the keyboard. | | `joystick` | Drive a DotBot from a joystick. | @@ -63,19 +63,21 @@ so it shares the controller's flags and serves the same dashboard. dotbot run simulator -w ``` -## `lh2-calibration` - capture & apply +## `lh2-calibration` - capture & apply (cabled) Lighthouse v2 calibration against a single serial-attached board. `collect` opens a TUI to capture LH2 counts; `apply` writes the saved calibration out as -a C header. +a C header. This is the cabled, bench path - for deployed bots, capture over the +air with [`swarm lh2-calibration`](swarm.md) instead. ```bash dotbot run lh2-calibration collect dotbot run lh2-calibration apply ./lh2_calibration.h ``` -See [the LH2 calibration guide](../guides/lh2-calibration.md). To push a -calibration to the whole fleet over the air, use [`swarm calibrate-lh2`](swarm.md). +See [the cabled LH2 calibration guide](../guides/lh2-calibration-cabled.md). To +capture without a cable, or to push a saved calibration to the fleet over the +air, use [`swarm lh2-calibration`](swarm.md). ## `demo` - built-in demos diff --git a/doc/cli/swarm.md b/doc/cli/swarm.md index f9e7cdfb..badaa5e4 100644 --- a/doc/cli/swarm.md +++ b/doc/cli/swarm.md @@ -46,8 +46,8 @@ The OTA payload is a **sandbox** app - a TrustZone non-secure `.bin`. Build it, or fetch a pre-compiled release: ```bash -dotbot fw artifacts --sandbox # builds -> ./artifacts/-sandbox-.bin -dotbot fw fetch -f 0.8.0rc1 # or pull from a release into ./artifacts// +dotbot fw artifacts --sandbox # builds -> ~/.dotbot/artifacts/dotbot-firmware-local/-sandbox-.bin +dotbot fw fetch # or pull the pinned releases into ~/.dotbot/artifacts/-/ ``` Sandbox apps include `dotbot`, `move`, `rgbled`, `spin`, `timer`. Artifact @@ -83,7 +83,7 @@ to override). If the broker needs auth, set `DOTBOT_MQTT_USER` / ```bash dotbot swarm status # who's out there + their state dotbot swarm status -w # keep watching -dotbot swarm flash ./artifacts/spin-sandbox-dotbot-v3.bin -ys +dotbot swarm flash ~/.dotbot/artifacts/dotbot-firmware-local/spin-sandbox-dotbot-v3.bin -ys dotbot swarm stop # back to bootloader (before re-flashing) dotbot swarm start # (re)start the loaded app dotbot swarm monitor # tail SWARMIT_EVENT_LOG from bots @@ -101,18 +101,24 @@ To replace a running experiment: `stop`, then `flash ... -ys`. | `-t`, `--ota-timeout` | seconds per OTA ACK (default `0.7`) | | `-r`, `--ota-max-retries` | retries per OTA message (default `10`) | -## 6. Push an LH2 calibration over the air +## 6. LH2 calibration over the air -Send a calibration (captured from one cabled bot - see -[LH2 calibration](../guides/lh2-calibration.md)) to the whole fleet: +Capture and push a Lighthouse-2 calibration for one DotBot without a cable, +driving it over the swarm. The arena geometry and `-d` sizing live in the +[LH2 calibration guide](../guides/lh2-calibration.md). ```bash -dotbot swarm stop -dotbot swarm calibrate-lh2 ~/.dotbot/calibration-.toml +dotbot swarm stop # capture only runs in READY +dotbot swarm lh2-calibration collect --device BC3D... -d 500 # capture from one bot -> solve -> save +dotbot swarm lh2-calibration push ~/.dotbot/calibration-.toml # apply to every ready bot ``` -It accepts a `calibration-*.toml` or the legacy raw payload; the format is -picked by file extension. +`collect` walks one bot through the four arena corners over the air, solves the +homography, and saves it under `~/.dotbot/`. `push` (no `--device`) then sends +that calibration to **every ready bot** - the arena shares one transform. +(`collect --push` is a single-bot shortcut: it sends only to the captured bot.) +`push` takes a `calibration-*.toml` or the legacy raw payload - the format is +picked by file extension. Get the `--device` address from `dotbot swarm status`. ## Two web servers - don't mix them up diff --git a/doc/conf.py b/doc/conf.py index 1c10a4a2..35b78e48 100644 --- a/doc/conf.py +++ b/doc/conf.py @@ -134,6 +134,10 @@ # intermittently time out or rate-limit the linkcheck bot. r"https://img\.shields\.io/", r"https://badge\.fury\.io/", + # dotbots.org and segger.com return 403 to the linkcheck bot (WAF / + # user-agent block); both links are valid for humans. + r"https?://www\.dotbots\.org", + r"https://www\.segger\.com/", ] # -- Options for autosummary/autodoc output ----------------------------------- diff --git a/doc/guides/index.md b/doc/guides/index.md index bd10c944..60fe7321 100644 --- a/doc/guides/index.md +++ b/doc/guides/index.md @@ -4,11 +4,20 @@ Task-oriented walkthroughs that span several commands. ```{toctree} :hidden: -lh2-calibration +simulator +one-bot controller +lh2-calibration +lh2-calibration-cabled ``` -- [Lighthouse 2 localization](lh2-calibration.md) - give your bots real-world - `(x, y)` positions. +- [Try it in the simulator](simulator.md) - run the full UI and script bots with + no hardware. +- [Drive a single DotBot](one-bot.md) - build, flash, and control one bot end to + end. - [Run the controller + web UI](controller.md) - drive and visualize a swarm from the browser. +- [Lighthouse 2 localization](lh2-calibration.md) - give your bots real-world + `(x, y)` positions, calibrated over the air. +- [LH2 calibration over a cable](lh2-calibration-cabled.md) - the bench + alternative, for a single USB-connected bot. diff --git a/doc/guides/lh2-calibration-cabled.md b/doc/guides/lh2-calibration-cabled.md new file mode 100644 index 00000000..9f20122c --- /dev/null +++ b/doc/guides/lh2-calibration-cabled.md @@ -0,0 +1,87 @@ +# LH2 calibration over a serial cable + +The bench alternative to the [over-the-air flow](lh2-calibration.md): calibrate a +single DotBot connected over USB, with no swarm provisioned. Reach for this when +you're working with one bot on the bench, or before the fleet is set up. For +already-deployed bots, prefer the over-the-air flow - no cable, no firmware swap. + +What LH2 calibration is, and the arena geometry (the `-d` square sizing), are +covered in the [main guide](lh2-calibration.md); this page is just the cabled +capture path. + +## Prerequisites + +- A DotBot v3 you can cable to your machine over USB-C (no external probe - the + v3 flashes over its on-board programmer). +- Two LH2 base stations facing the arena, and a square marked on the floor (see + [Sizing `-d`](lh2-calibration.md) in the main guide). +- The `[calibrate]` extra: + +```bash +pip install --pre 'pydotbot[calibrate]' +``` + +## 1. Flash the capture firmware + +The `lh2_calibration` app streams raw LH2 counts over serial. Flash it to the +cabled bot (see [device](../cli/device.md) for serial-prefix selection): + +```bash +dotbot device flash lh2_calibration -s 77 # board defaults to dotbot-v3 +``` + +## 2. Capture the four reference points + +Place the bot on the floor square and run the TUI. `-d` is the side length of +the square, in millimeters: + +```bash +dotbot run lh2-calibration collect -p /dev/cu.usbmodem... -d 500 +``` + +Move the bot to each corner - Top left -> Top right -> Bottom left -> Bottom +right - pressing the matching button in the TUI at each. When all four are +captured, save. The calibration is written under `~/.dotbot/` (a +`calibration-.toml`), the same place the over-the-air flow uses. + +| Flag | Default | Meaning | +|---|---|---| +| `-p`, `--port` | auto-detect | Serial port of the calibration firmware. | +| `-d`, `--distance` | calibration default | Square side length, **in mm**. | +| `-n`, `--extra-lh-num` | `0` | Extra base stations beyond the first (0–5). | +| `--input-data` | - | Re-process a saved capture instead of capturing live. | + +See `dotbot run lh2-calibration collect --help` for the full list. + +## 3. Use the calibration + +Send the saved calibration to the fleet over the air (the same command the +over-the-air flow uses - stop any running app first): + +```bash +dotbot swarm stop +dotbot swarm lh2-calibration push ~/.dotbot/calibration-.toml +``` + +### Bake it into the bootloader (header path) + +For a fresh board whose bootloader bakes the calibration in at compile time +(rather than receiving it over the air), export the saved calibration as a C +header instead: + +```bash +dotbot run lh2-calibration apply ./lh2_calibration.h +``` + +The swarmit secure bootloader `#include`s this file; rebuild and reflash the +bootloader for it to take effect. For already-running bots, prefer the +over-the-air push above - no reflash needed. + +## Troubleshooting + +- **No counts in the TUI** - wrong `-p` port, or the bot can't see both base + stations. Confirm line-of-sight and that the base-station LEDs are steady. +- **Positions look skewed or mirrored** - the corners were captured out of + order. Re-run `collect` and follow TL -> TR -> BL -> BR exactly. +- **Positions are scaled wrong** - `-d` didn't match the real square. It's in + millimeters, not centimeters. diff --git a/doc/guides/lh2-calibration.md b/doc/guides/lh2-calibration.md index 647b7083..f39eb36d 100644 --- a/doc/guides/lh2-calibration.md +++ b/doc/guides/lh2-calibration.md @@ -3,58 +3,82 @@ Lighthouse 2 gives every DotBot a real-world **(x, y) position** on your arena floor. Two SteamVR base stations sweep the room with IR; each bot's LH2 sensor times the sweeps. Calibration is the one-time step that maps those raw sweep -counts to metric coordinates: you place one bot on four known points of a +counts to metric coordinates: you place one bot on four known corners of a square, capture, and the resulting transform is pushed to the whole fleet. -You do this once per physical setup (move a base station → recalibrate). +You do this once per physical setup (move a base station -> recalibrate). + +**The default flow is over the air** - drive one already-deployed bot through the +corners over the swarm, no cable and no firmware swap. If you'd rather calibrate +a single bot on the bench over USB, see +[LH2 calibration over a cable](lh2-calibration-cabled.md). ## Prerequisites -- A DotBot v3 you can cable to your machine over USB-C (no external probe - needed - the v3 flashes over its on-board programmer). +- A provisioned swarm: a gateway plus sandbox-host bots, reachable from your + config (see [swarm](../cli/swarm.md)). - Two LH2 base stations mounted ~2 m up, facing the arena. -- A square marked on the floor with a known side length, plus the - `[calibrate]` extra installed: +- A square marked on the floor with a known side length. +- The `[calibrate]` extra (the homography solve uses opencv): ```bash pip install --pre 'pydotbot[calibrate]' ``` -## 1. Flash the capture firmware +## Capture and push -The `lh2_calibration` app streams raw LH2 counts over serial. Flash it to the -cabled bot (see [device](../cli/device.md) for serial-prefix selection): +The calibration is a property of the **arena** (the base-station layout), not the +individual bot, so you capture once from any one bot and apply the result to the +whole fleet. Two steps: ```bash -dotbot device flash lh2_calibration -s 77 # board defaults to dotbot-v3 +dotbot swarm stop # put the bots in READY +dotbot swarm lh2-calibration collect \ + --device BC3D3C8A2A6F8E68 -d 500 # capture from one bot -> solve -> save +dotbot swarm lh2-calibration push \ + ~/.dotbot/calibration-.toml # apply to every ready bot ``` -## 2. Capture the four reference points - -Place the bot on the floor square and run the TUI. `-d` is the **side length of -the square, in millimeters**: - -```bash -dotbot run lh2-calibration collect -p /dev/cu.usbmodem... -d 500 +`collect` walks one bot through the four corners - **top-left -> top-right -> +bottom-left -> bottom-right** - (capture only runs while the bot is in READY, so +`swarm stop` first). Each prompt triggers a raw-count capture over the air; it +then solves the homography and saves a `calibration-.toml` under `~/.dotbot/` +(the path is printed at the end). Find the `--device` address with +`dotbot swarm status`. + +`push` with **no `--device`** sends that calibration to **every ready bot** - the +whole arena shares one transform. It accepts a `calibration-*.toml` or the legacy +raw `calibration.out` payload; the format is picked by file extension. + +```{note} +`collect --push` is a **single-bot shortcut**: it sends the result to *only* the +`--device` bot you captured from (handy to spot-check that one bot, or for a +single-bot setup). To calibrate the fleet, run the standalone `push` above - it +targets all ready bots. ``` -Move the bot to each corner - Top left → Top right → Bottom left → Bottom right -- pressing the matching button in the TUI at each. When all four are captured, -save. The calibration is written under `~/.dotbot/` (a `calibration-.toml`). +Once pushed, the bots report positions, which show up live in the +[controller](../cli/run.md) Web UI. -Common `collect` flags: +### `collect` flags | Flag | Default | Meaning | |---|---|---| -| `-p`, `--port` | auto-detect | Serial port of the calibration firmware. | -| `-d`, `--distance` | - | Square side length, **in mm** (see sizing below). | -| `-n`, `--extra-lh-num` | `0` | Extra base stations beyond the first (0–5). | -| `--input-data` | - | Re-process a saved capture instead of capturing live. | +| `--device` | (required) | DotBot link-layer address in hex (from `dotbot swarm status`). | +| `-d`, `--distance` | calibration default | Square side length, **in mm** (see sizing below). | +| `-n`, `--conn` / `-s`, `--swarm-id` | from config | Swarm connection, like the other `dotbot swarm` commands. | +| `--timeout` | `5` s | Seconds to wait for each capture before re-triggering. | +| `--retries` | `3` | Re-trigger this many times per corner before giving up. | +| `--tag` | - | Arena/setup label (e.g. `office-2x2m`) added to the filename + metadata. | +| `--push` | off | After solving, send to the captured `--device` bot **only** (use the standalone `push` for the whole fleet). | -See `dotbot run lh2-calibration collect --help` for the full list. +See `dotbot swarm lh2-calibration collect --help` for the full list. -**Sizing `-d`** - the usable arena is **5× the square side**, with the square -centered (a `2·d` margin on every side): +## Sizing `-d` + +`-d` is the **side of your reference square, in millimeters**. The usable arena +is **5× the square side**, with the square centered (a `2·d` margin on every +side): ``` ←─────────────── 5·d ────────────→ @@ -82,42 +106,12 @@ square side (`--distance`, in mm), `5·d` the resulting arena. | `500` | 50 cm | 2.5 m × 2.5 m | | `800` | 80 cm | 4.0 m × 4.0 m (used for the 725-bot Limerick run) | -## 3. Push the calibration to the fleet - -Send the captured calibration over the air. Stop any running app first, then -push the `.toml` (see [swarm](../cli/swarm.md) for the connection config): - -```bash -dotbot swarm stop -dotbot swarm calibrate-lh2 ~/.dotbot/calibration-.toml -``` - -`calibrate-lh2` accepts either a `calibration-*.toml` or the legacy raw -`calibration.out` payload - the format is picked by file extension. - -Once pushed, the bots report positions, which show up live in the -[controller](../cli/run.md) Web UI. - -## First-time flashing (header path) - -For a fresh board whose bootloader bakes the calibration in at compile time -(rather than receiving it over the air), export the saved calibration as a C -header instead: - -```bash -dotbot run lh2-calibration apply ./lh2_calibration.h -``` - -The swarmit secure bootloader `#include`s this file; rebuild and reflash the -bootloader for it to take effect. For already-running bots, prefer the OTA path -in step 3 - no reflash needed. - ## Troubleshooting -- **No counts in the TUI** - wrong `-p` port, or the bot can't see both base - stations. Confirm line-of-sight and that the LEDs on the base stations are - steady. +- **Capture times out** - the bot isn't in READY (run `dotbot swarm stop` + first), or it can't see both base stations. The address passed to `--device` + must match one from `dotbot swarm status`. - **Positions look skewed or mirrored** - the corners were captured out of - order. Re-run `collect` and follow TL → TR → BL → BR exactly. + order. Re-run `collect` and follow TL -> TR -> BL -> BR exactly. - **Positions are scaled wrong** - `-d` didn't match the real square. It's in millimeters, not centimeters. diff --git a/doc/guides/one-bot.md b/doc/guides/one-bot.md new file mode 100644 index 00000000..7b059608 --- /dev/null +++ b/doc/guides/one-bot.md @@ -0,0 +1,53 @@ +# Drive a single DotBot + +Build and flash one DotBot and a gateway, cable them to your computer, and drive +the bot from the web UI. This is the smallest real-hardware setup. For the +no-hardware path, use the [simulator](simulator.md) instead. + +Building firmware needs SEGGER Embedded Studio and `nrfjprog` (see the README +prerequisites). To skip building, fetch a pre-built release with `dotbot fw +fetch -f ` and flash those artifacts instead. + +## 1. Build and flash the DotBot + +The DotBot v3 is an nRF5340, which has two cores - the application core (your +app) and the network core (the radio) - so you build and flash two images: + +```bash +# build the bare dotbot apps into the cache (needs SEGGER Embedded Studio) +dotbot fw artifacts --app dotbot +dotbot fw artifacts --app nrf5340_net --target nrf5340dk-net +# cable-flash to the bot whose J-Link serial starts with 77 +dotbot device flash dotbot -s 77 # app core +dotbot device flash nrf5340_net -b nrf5340dk-net -s 77 # network core +``` + +## 2. Build and flash the gateway + +The gateway is a dev board (e.g. an nRF52840-DK) plugged into your computer; it +bridges the robot's radio to USB serial. + +```bash +dotbot fw artifacts --app dotbot_gateway --target nrf52840dk +# cable-flash to the DK whose J-Link serial starts with 10 +dotbot device flash dotbot_gateway -b nrf52840dk -s 10 +``` + +## 3. Drive it from the web UI + +With the gateway plugged in, point the controller at its serial port and open +the web UI: + +```bash +dotbot run controller --conn /dev/ttyACM0 -w # serial gateway; no swarm-id needed +``` + +Select the bot in the browser and steer it with the joystick. See the +[controller + web UI guide](controller.md) for the full UI tour, and +[`fw`](../cli/fw.md) / [`device`](../cli/device.md) for the firmware commands. + +## Next + +- Operate many bots over the air - the [`swarm`](../cli/swarm.md) reference. +- Add real-world positions - [LH2 calibration over a cable](lh2-calibration-cabled.md) + (or [over the air](lh2-calibration.md) once you've provisioned a swarm). diff --git a/doc/guides/simulator.md b/doc/guides/simulator.md new file mode 100644 index 00000000..75b1e717 --- /dev/null +++ b/doc/guides/simulator.md @@ -0,0 +1,66 @@ +# Try it in the simulator + +The simulator runs the **full controller and web UI with no hardware** - no bot, +no gateway, no radio. It's the fastest way to see DotBot work, and because it +exposes the exact same REST/WebSocket API as the real controller, code you write +against the simulator runs unchanged against real robots. + +## Start it + +```bash +dotbot run simulator -w +``` + +This opens the web UI at driving a simulated +swarm. `dotbot run simulator` is shorthand for +`dotbot run controller --conn simulator`, so everything in the +[controller + web UI guide](controller.md) applies. Drive the bots from the +joystick and watch them on the map. + +## Drive one bot in a circle + +The simplest demo - it grabs the first bot the controller sees and drives it in a +circle. With the simulator running, in a second terminal: + +```bash +dotbot run demo circle +``` + +## Drive it from your own code + +That demo is ~15 lines talking to the controller's REST API over +[`requests`](https://pypi.org/project/requests/) (bundled with pydotbot) - here +is the whole thing, a template for your own scripts: + +```python +import requests, time + +BASE = "http://localhost:8000" +bot = requests.get(f"{BASE}/controller/dotbots").json()[0]["address"] + +# roll in a circle for ~5 s - left_y and right_y are the two wheel speeds +for _ in range(50): + requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", + json={"left_x": 0, "left_y": 60, "right_x": 0, "right_y": 30}) + time.sleep(0.1) +requests.put(f"{BASE}/controller/dotbots/{bot}/0/move_raw", + json={"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}) +``` + +The full surface - every endpoint, the live WebSocket stream, and CSV data +logging - is in the [REST / WebSocket reference](../reference/rest.md) (or the +[MQTT bridge](../reference/mqtt.md)). A higher-level Python SDK is planned; today +you talk to the controller over REST/WebSocket/MQTT. + +## More examples + +```bash +dotbot run demo --list # what's available +dotbot run demo qr # phone-as-joystick over QrKey +``` + +Richer multi-bot scenarios - work-and-charge, charging-station, labyrinth, the +naming game, motion shapes - live in `dotbot/examples/`, each with its own +README and a simulator init state. They drive the controller over the same +REST/WebSocket API shown above, so they run against the simulator or real +hardware unchanged. diff --git a/doc/index.md b/doc/index.md index 6c0f5b9b..fdcce713 100644 --- a/doc/index.md +++ b/doc/index.md @@ -15,12 +15,10 @@ New here? DotBots are small wheeled robots you drive from your browser or your own code - one bot, or a swarm of hundreds. Pick a starting point: - **Try it with no hardware** - the simulator runs the full web UI with no bot - or gateway needed: `dotbot run simulator -w`. Then explore the - [web-UI guide](guides/controller.md). + or gateway needed: `dotbot run simulator -w`. See the + [simulator guide](guides/simulator.md). - **Get one bot moving** - build and cable-flash a single DotBot and gateway, - then drive it from the browser. See the one-bot quickstart below - ([`fw`](cli/fw.md) / [`device`](cli/device.md) / - [controller guide](guides/controller.md)). + then drive it from the browser. See the [one-bot guide](guides/one-bot.md). - **Run a swarm experiment** - provision and command many bots over the air. The swarm quickstart below is the main path; then see [`swarm`](cli/swarm.md) and [LH2 localization](guides/lh2-calibration.md). diff --git a/doc/reference/configuration.md b/doc/reference/configuration.md index f57c8f44..ead8a3ea 100644 --- a/doc/reference/configuration.md +++ b/doc/reference/configuration.md @@ -81,7 +81,6 @@ Set once at the top of the file; any section or deployment can override them. | `conn` | Default connection string (`mqtts://host:port`, a serial path, or `simulator`). | | `swarm_id` | Swarm id selecting the MQTT topic namespace. | | `log_level` | Logging verbosity. | -| `artifacts_dir` | Where firmware artifacts are read/written. | | `default_deployment` | Name of the deployment to select when neither `--deployment` nor `DOTBOT_DEPLOYMENT` is given. | ## Section tables @@ -220,7 +219,6 @@ default_deployment = "inria" # used when --deployment / DOTBOT_D conn = "mqtts://broker.local:8883" swarm_id = "0001" log_level = "info" -artifacts_dir = "./artifacts" # A physical deployment. Select it with `--deployment inria`, DOTBOT_DEPLOYMENT, or # default_deployment above - don't edit this table to switch deployments. diff --git a/dotbot/cli/_artifacts.py b/dotbot/cli/_artifacts.py index dab10222..b5516a78 100644 --- a/dotbot/cli/_artifacts.py +++ b/dotbot/cli/_artifacts.py @@ -3,9 +3,9 @@ """Shared artifact-resolution + friendly-error helpers for `fw` / `device`. -Owns the CWD-local ``./artifacts/`` convention, the absolute-path echo on -every cache read/write (so running from a random directory never silently -touches a relative path the user didn't name), the auto-resolve decision +Owns the user-level ``~/.dotbot/artifacts/`` cache convention, the +absolute-path echo on every cache read/write (so you always see where files +landed, regardless of the directory you ran from), the auto-resolve decision tree used by ``dotbot device flash `` (present → build → error), and the two centralized tool-missing messages (SES for builds, nrfjprog for device ops). @@ -15,19 +15,32 @@ SES nor a firmware repo) stays cheap and side-effect-free. """ +import os from pathlib import Path import click +# Single source of truth for the default cache location, so changing it is a +# one-line edit. `artifacts_dir()` is the behavioral path (env-overridable, ~ +# expanded, resolved); `DEFAULT_ARTIFACTS_DISPLAY` is the human-readable form +# for help text. (Docstrings and the Markdown docs can't interpolate a +# variable, so they spell the path out literally - keep them in sync by hand.) +_DEFAULT_ARTIFACTS_PARTS = (".dotbot", "artifacts") +DEFAULT_ARTIFACTS_DISPLAY = "~/" + "/".join(_DEFAULT_ARTIFACTS_PARTS) + def artifacts_dir() -> Path: - """The CWD-local ``./artifacts/`` directory, resolved absolute. + """The firmware cache: ``~/.dotbot/artifacts/`` by default. - The single source of truth for where build outputs land and fetched - releases are cached. Per-workspace (no global ``~/.dotbot/fw``), so - two checkouts never collide. + User-level and shared across working directories (like other tools cache + downloaded firmware), so you don't re-download a release per directory and + the launch dir stays clean. Override with ``$DOTBOT_ARTIFACTS_DIR``. """ - return (Path.cwd() / "artifacts").resolve() + override = os.environ.get("DOTBOT_ARTIFACTS_DIR") + base = ( + Path(override) if override else Path.home().joinpath(*_DEFAULT_ARTIFACTS_PARTS) + ) + return base.expanduser().resolve() def echo_artifact_path(path: Path, *, action: str = "using") -> None: @@ -59,6 +72,36 @@ def ensure_nrfjprog() -> None: raise friendly_nrfjprog_error() +def _find_in_cache(name: str) -> Path | None: + """Find a firmware file across the source-qualified cache dirs. + + The cache holds ``-/`` (fetched releases) and + ``-local[-link]/`` (built/linked) subdirs. Prefer a local build + (you're iterating on it), else the newest-versioned release dir. + """ + root = artifacts_dir() + if not root.is_dir(): + return None + local_hits: list[Path] = [] + released_hits: list[Path] = [] + for sub in sorted(root.iterdir()): + if not sub.is_dir(): + continue + candidate = sub / name + if candidate.is_file(): + if sub.name.endswith(("-local", "-local-link")): + local_hits.append(candidate) + else: + released_hits.append(candidate) + if local_hits: + return local_hits[0] + if released_hits: + # newest by dir name (a coarse version sort; good enough for picking + # the latest release of a given app) + return sorted(released_hits, key=lambda p: p.parent.name, reverse=True)[0] + return None + + def resolve_app_artifact( app: str, *, @@ -68,20 +111,20 @@ def resolve_app_artifact( ) -> Path: """Auto-resolve a single app's firmware artifact for cable-flashing. - Decision tree (npm-style): present in ``./artifacts/`` → build from - source → clear error pointing at build/fetch. An *explicit file path* + Decision tree (npm-style): present in ``~/.dotbot/artifacts/`` → build + from source → clear error pointing at build/fetch. An *explicit file path* is handled by the caller before this is reached. - - Flat ``./artifacts/-.hex`` (bare) or - ``./artifacts/-sandbox-.bin`` (sandbox), as produced by - `dotbot fw artifacts`. + - Flat ``-.hex`` (bare) or ``-sandbox-.bin`` + (sandbox), as produced by `dotbot fw artifacts`, found across the + cache's ``-/`` and ``-local/`` dirs. - Else, if a DotBot-firmware repo is locatable, build it (needs SES) and use the SES output path. - Else, a friendly error telling the user to build or fetch first. """ name = f"{app}-sandbox-{board}.bin" if sandbox else f"{app}-{board}.hex" - cached = artifacts_dir() / name - if cached.is_file(): + cached = _find_in_cache(name) + if cached is not None: echo_artifact_path(cached, action="using") return cached @@ -114,6 +157,6 @@ def resolve_app_artifact( "DotBot-firmware source to build from.\n" " • `dotbot fw build " f"-a {app} -t {board}{' --sandbox' if sandbox else ''}` to build, or\n" - " • `dotbot fw fetch -f ` to download a release, then retry, or\n" - " • pass an explicit path: `dotbot device flash ./artifacts/`." + " • `dotbot fw fetch` to download the latest releases, then retry, or\n" + " • pass an explicit path: `dotbot device flash `." ) diff --git a/dotbot/cli/demo.py b/dotbot/cli/demo.py index 940bdeee..ff78543c 100644 --- a/dotbot/cli/demo.py +++ b/dotbot/cli/demo.py @@ -12,6 +12,7 @@ import click +from dotbot.examples.circle.circle import main as _circle_main from dotbot.examples.qrkey_demo.cli import main as _qrkey_main @@ -43,4 +44,5 @@ def cmd(ctx, list_demos): # We pass `name=...` rather than mutating `_qrkey_main.name` so the # demo's own test suite (which imports the same Click command) stays # unaffected. +cmd.add_command(_circle_main, name="circle") cmd.add_command(_qrkey_main, name="qr") diff --git a/dotbot/cli/device.py b/dotbot/cli/device.py index 954a5d11..f0b92323 100644 --- a/dotbot/cli/device.py +++ b/dotbot/cli/device.py @@ -19,6 +19,7 @@ import click from dotbot.cli._artifacts import ( + DEFAULT_ARTIFACTS_DISPLAY, artifacts_dir, ensure_nrfjprog, resolve_app_artifact, @@ -54,7 +55,7 @@ def _looks_like_path(value: str) -> bool: show_default=True, help=( "Target board: selects the chip family + core to flash (nRF52 vs " - "nRF5340 app/net) and resolves - in ./artifacts/." + f"nRF5340 app/net) and resolves - in {DEFAULT_ARTIFACTS_DISPLAY}/." ), ) @click.option("--sandbox", is_flag=True, help="Resolve the sandbox-app flavor (.bin).") @@ -70,7 +71,7 @@ def _looks_like_path(value: str) -> bool: def flash(ctx, app, sn_starting_digits, board, sandbox, config): """Flash a firmware image to one cabled device (whole-chip program). - APP is an app name (resolved against ./artifacts/, building from source + APP is an app name (resolved against ~/.dotbot/artifacts/, building from source if needed) or an explicit `.hex`/`.bin` file path. `--board` selects the chip family + core to program (see `dotbot fw targets`); no sandbox host is required. @@ -96,10 +97,10 @@ def _fw_version_option(f): return click.option( "--fw-version", "-f", - required=True, + default=None, help=( - "Release version to flash, e.g. 0.8.0rc1. Its binaries are " - "fetched into ./artifacts/ if not already cached." + "Release version to flash, e.g. 0.8.0rc2 (default: the swarmit " + f"version pydotbot pins). Binaries are fetched into {DEFAULT_ARTIFACTS_DISPLAY}/ if not cached." ), )(f) @@ -133,8 +134,9 @@ def flash_swarmit_sandbox( Flashes the SwarmIT bootloader (app core) + netcore + writes the network identity. Auto-fetches the release if not already in - ./artifacts//. + ~/.dotbot/artifacts/swarmit-/. """ + from dotbot.firmware.fetch import pinned_version from dotbot.firmware.flash import flash_role, normalize_network_id swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) @@ -143,6 +145,11 @@ def flash_swarmit_sandbox( "no swarm id: pass --swarm-id, or set swarm_id (or a " "deployment) in your config." ) + if fw_version is None: + fw_version = pinned_version("swarmit") + click.echo( + f"No version specified, using the pinned swarmit version: {fw_version}" + ) ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( @@ -171,6 +178,7 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): identity. Auto-fetches the release if absent. (To run the host-side UART<->MQTT bridge instead, use `dotbot run gateway`.) """ + from dotbot.firmware.fetch import pinned_version from dotbot.firmware.flash import flash_role, normalize_network_id swarm_id = from_config(ctx, "swarm_id", "swarm_id", None) @@ -179,6 +187,11 @@ def flash_mari_gateway(ctx, swarm_id, fw_version, sn_starting_digits): "no swarm id: pass --swarm-id, or set swarm_id (or a " "deployment) in your config." ) + if fw_version is None: + fw_version = pinned_version("swarmit") + click.echo( + f"No version specified, using the pinned swarmit version: {fw_version}" + ) ensure_nrfjprog() net_id = normalize_network_id(swarm_id) flash_role( diff --git a/dotbot/cli/fw.py b/dotbot/cli/fw.py index df927cdf..a1d4966e 100644 --- a/dotbot/cli/fw.py +++ b/dotbot/cli/fw.py @@ -9,22 +9,24 @@ - `build` compiles from source via SES (`emBuild`) in `DotBot-firmware`, leaving the result in the SES `Output/.../Exe/` tree and echoing that - path — it does *not* copy into `./artifacts/`. Bare apps by default; + path — it does *not* copy into the cache. Bare apps by default; `--sandbox` builds the TrustZone NS flavor (`sandbox-`, `.bin`). -- `artifacts` builds *and* collects the result into `./artifacts/`, with - the flat `-.hex` / `-sandbox-.bin` names. -- `fetch` downloads a published release into `./artifacts//`. -- `list` shows what's cached in `./artifacts/`. +- `artifacts` builds *and* collects the result into the cache + (`~/.dotbot/artifacts/dotbot-firmware-local/`), with the flat + `-.hex` / `-sandbox-.bin` names. +- `fetch` downloads the pinned release (or a `-f `/`latest` + override) into `~/.dotbot/artifacts/-/`. +- `list` shows what's cached in `~/.dotbot/artifacts/`. - `make` is the low-level escape hatch: it forwards arbitrary arguments to `make` in the firmware repo (workspace-resolved SEGGER_DIR) for the Makefile knobs `build` deliberately doesn't model. -Only `artifacts` and `fetch` populate `./artifacts/`. The device-flash +Only `artifacts` and `fetch` populate the cache. The device-flash commands then auto-resolve their input, by *different* rules: `dotbot -device flash ` resolves an app image present-in-`./artifacts/` → +device flash ` resolves an app image present in `~/.dotbot/artifacts/` → build-from-source → error (it never fetches); `device flash-swarmit-sandbox` -/ `flash-mari-gateway` resolve a release's system firmware -present-in-`./artifacts/` → fetch (they never build). +/ `flash-mari-gateway` resolve a release's system firmware present in +`~/.dotbot/artifacts/` → fetch (they never build). """ import sys @@ -32,7 +34,11 @@ import click -from dotbot.cli._artifacts import artifacts_dir, echo_artifact_path +from dotbot.cli._artifacts import ( + DEFAULT_ARTIFACTS_DISPLAY, + artifacts_dir, + echo_artifact_path, +) from dotbot.cli._cfg import from_config from dotbot.cli._fw_helpers import ( BARE_TARGETS, @@ -211,7 +217,7 @@ def list_targets(sandbox): "out_dir", type=click.Path(file_okay=False, dir_okay=True), default=None, - help="Where to collect artifacts. Default: ./artifacts/ (your CWD).", + help=f"Where to collect artifacts. Default: {DEFAULT_ARTIFACTS_DISPLAY}/dotbot-firmware-local/.", ) @click.option( "--print-path", @@ -222,7 +228,12 @@ def list_targets(sandbox): @click.option("-v", "--verbose", is_flag=True, default=False) @click.pass_context def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbose): - """Build + collect artifacts into ./artifacts/ (default).""" + """Build + collect artifacts into the local cache (default). + + Without --out, built apps land in the source-qualified + ``/dotbot-firmware-local/`` dir so `device flash ` finds them + alongside fetched releases. + """ import shutil target = from_config(ctx, "target", "board", "fw") @@ -237,7 +248,11 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos ) click.echo(str(artifact_path(build_target, project, config))) return - out = Path(out_dir).resolve() if out_dir else artifacts_dir() + out = ( + Path(out_dir).resolve() + if out_dir + else artifacts_dir() / "dotbot-firmware-local" + ) click.echo( f"Building + collecting artifacts for {target} ({config}) → {out}/...", err=True, @@ -263,28 +278,73 @@ def artifacts(ctx, target, project, config, sandbox, out_dir, print_path, verbos @cmd.command() @click.option( - "--fw-version", "-f", required=True, help="Release version tag, or 'local'." + "--source", + "-S", + type=click.Choice(list(("swarmit", "dotbot-firmware"))), + default=None, + help="Limit to one source (default: fetch the pinned version from all sources).", +) +@click.option( + "--fw-version", + "-f", + default=None, + help="Override the pinned version for --source: a release tag, 'latest', or 'local'.", ) @click.option( "--local-root", type=click.Path(path_type=Path, file_okay=False, dir_okay=True), - help="Root of a local DotBot-firmware/swarmit build (with --fw-version local).", + help="Root of a local build tree (with --source --fw-version local).", ) -def fetch(fw_version, local_root): - """Download a released firmware set into ./artifacts//.""" - from dotbot.firmware.flash import fetch_assets +def fetch(source, fw_version, local_root): + """Download firmware into ~/.dotbot/artifacts/-/. + + With no flags, fetches the exact release this pydotbot is pinned to, from + every source: swarmit (swarm system images, version inferred from the + installed swarmit package) and DotBot-firmware (bare + sandbox apps, the + version pydotbot is tested against). The two version independently, so + overriding with -f requires a --source - pass `-f latest` for the newest + release or `-f ` for a specific one. + """ + from dotbot import pydotbot_version + from dotbot.firmware.fetch import ( + DEFAULT_FETCH_SOURCES, + _short_path, + fetch_assets, + pinned_version, + ) - out = fetch_assets(fw_version, artifacts_dir(), local_root) - echo_artifact_path(out, action="fetched into") + if fw_version is not None and source is None: + raise click.ClickException( + "Pass --source with -f/--fw-version: swarmit and dotbot-firmware " + "version independently." + ) + sources = [source] if source else list(DEFAULT_FETCH_SOURCES) + fetched: list[Path] = [] + for src in sources: + if fw_version is None: + version = pinned_version(src) + click.echo( + f"Fetching {src} {version} (pinned by pydotbot {pydotbot_version()})..." + ) + else: + version = fw_version + if version == "latest": + click.echo(f"Fetching the latest {src} release...") + fetched.append(fetch_assets(src, version, artifacts_dir(), local_root)) + click.echo("\nDone. Firmware fetched into:") + for path in fetched: + click.echo(f" {_short_path(path)}") @cmd.command(name="list") def list_artifacts(): - """List firmware artifacts cached in ./artifacts/.""" + """List firmware artifacts cached in ~/.dotbot/artifacts/.""" root = artifacts_dir() echo_artifact_path(root, action="listing") if not root.is_dir(): - click.echo("(no ./artifacts/ yet — run `dotbot fw build` or `dotbot fw fetch`)") + click.echo( + "(nothing cached yet — run `dotbot fw fetch` or `dotbot fw artifacts`)" + ) return found = sorted( p for p in root.rglob("*") if p.is_file() and p.suffix in (".hex", ".bin") diff --git a/dotbot/cli/main.py b/dotbot/cli/main.py index e7a3ba81..8fca3951 100644 --- a/dotbot/cli/main.py +++ b/dotbot/cli/main.py @@ -5,7 +5,7 @@ The top level is the four object-namespaces, each one *kind of thing*: - fw — firmware artifacts (files in ./artifacts/, no hardware) + fw — firmware artifacts (cached in ~/.dotbot/artifacts/, no hardware) device — one connected device (cable / probe) swarm — the fleet (radio / OTA) run — host-side processes (software you launch on your computer) @@ -128,6 +128,11 @@ def cli(ctx, config_path, deployment_name): except ConfigError as exc: raise click.ClickException(str(exc)) from exc + if path is not None: + click.echo(f"using config file at {path}", err=True) + else: + click.echo("no config file found; using built-in defaults", err=True) + ctx.obj["config"] = config ctx.obj["config_path"] = path ctx.obj["deployment"] = deployment diff --git a/dotbot/config.py b/dotbot/config.py index 9e76a20b..250d048a 100644 --- a/dotbot/config.py +++ b/dotbot/config.py @@ -163,7 +163,6 @@ class DotbotConfig(_Strict): """The whole file: top-level shared keys + the four section tables + deployments.""" default_deployment: str | None = None - artifacts_dir: str | None = None log_level: str | None = None conn: Conn = None swarm_id: str | None = None diff --git a/dotbot/examples/circle/__init__.py b/dotbot/examples/circle/__init__.py new file mode 100644 index 00000000..2eb0e0b9 --- /dev/null +++ b/dotbot/examples/circle/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: 2026-present Inria +# SPDX-License-Identifier: BSD-3-Clause diff --git a/dotbot/examples/circle/circle.py b/dotbot/examples/circle/circle.py new file mode 100644 index 00000000..0cd53ca8 --- /dev/null +++ b/dotbot/examples/circle/circle.py @@ -0,0 +1,61 @@ +# SPDX-FileCopyrightText: 2026-present Inria +# SPDX-License-Identifier: BSD-3-Clause + +"""The simplest DotBot demo: drive one bot in a circle over the REST API. + +No pydotbot internals - just HTTP with `requests`, so it doubles as a +copy-pasteable template for your own scripts. Start a controller +(e.g. `dotbot run simulator -w`), then run `dotbot run demo circle`. +""" + +import time + +import click +import requests + + +@click.command(name="circle", help="Drive one bot in a circle (bare-REST demo).") +@click.option( + "--base", + default="http://localhost:8000", + show_default=True, + help="Controller REST base URL.", +) +@click.option( + "--seconds", + "-t", + default=5.0, + show_default=True, + help="How long to drive, in seconds.", +) +def main(base, seconds): + """Drive the first DotBot the controller sees in a circle, then stop.""" + try: + bots = requests.get(f"{base}/controller/dotbots", timeout=5).json() + except requests.RequestException as exc: + raise click.ClickException( + f"Cannot reach a controller at {base} ({exc}). " + "Start one first, e.g. `dotbot run simulator -w`." + ) from exc + if not bots: + raise click.ClickException( + f"No DotBots at {base}. The simulator (`dotbot run simulator -w`) " + "spawns one; on hardware, flash and connect a bot first." + ) + + address = bots[0]["address"] + move = f"{base}/controller/dotbots/{address}/0/move_raw" + click.echo(f"Driving {address} in a circle for {seconds:.0f}s (Ctrl-C to stop)...") + + # left_y and right_y are the two wheel speeds; unequal -> the bot turns. + end = time.time() + seconds + try: + while time.time() < end: + requests.put( + move, json={"left_x": 0, "left_y": 60, "right_x": 0, "right_y": 30} + ) + time.sleep(0.1) + finally: + # always stop the motors, even on Ctrl-C + requests.put(move, json={"left_x": 0, "left_y": 0, "right_x": 0, "right_y": 0}) + click.echo("Done.") diff --git a/dotbot/firmware/fetch.py b/dotbot/firmware/fetch.py new file mode 100644 index 00000000..6c13eca3 --- /dev/null +++ b/dotbot/firmware/fetch.py @@ -0,0 +1,302 @@ +"""Firmware release fetching + cache layout (no CLI, no flashing). + +The engine behind `dotbot fw fetch` and the auto-fetch hook in +`flash.flash_role`: decide which GitHub release to pull (pinned / latest / +an explicit tag), download every asset it publishes into the +source-qualified cache (`~/.dotbot/artifacts/-/`), and +record provenance in a `manifest.json`. Pure library code; the Click +surface lives in `dotbot/cli/fw.py`. + +Kept separate from `flash.py` (the hardware-facing flashing engine): this +module never touches a device, only the network + the cache. +""" + +from __future__ import annotations + +import json +import os +import shutil +import time +import urllib.error +import urllib.request +from concurrent.futures import ThreadPoolExecutor, as_completed +from datetime import datetime, timezone +from pathlib import Path + +import click + +GITHUB_API = "https://api.github.com/repos" +# Firmware release sources. swarmit ships the swarm system images (bootloader +# + mari netcore + mari gateway); DotBot-firmware ships the bare apps (.hex) +# and the sandbox apps (.bin). Each is cached in its own -/ +# subdir of the artifacts cache so versions and provenance never collide. +RELEASE_SOURCES = { + "swarmit": "DotBots/swarmit", + "dotbot-firmware": "DotBots/DotBot-firmware", +} +DEFAULT_FETCH_SOURCES = ("swarmit", "dotbot-firmware") +# The DotBot-firmware release this pydotbot is built and tested against. Unlike +# swarmit (a Python dependency, whose firmware release is tagged identically to +# the package, so we read it from importlib.metadata), DotBot-firmware is not a +# Python package - so the expected version is declared here and bumped +# deliberately when pydotbot adopts a new release. `dotbot fw fetch` (no -f) +# pulls exactly this; -f overrides it. +DOTBOT_FIRMWARE_VERSION = "1.22.0rc1" + +# Transient HTTP statuses worth retrying (GitHub's asset CDN 502s now and +# then under concurrent load; 429 is rate-limiting). +_RETRY_STATUS = {429, 500, 502, 503, 504} + + +def resolve_fw_root(bin_dir: Path, source: str, fw_version: str) -> Path: + return bin_dir / f"{source}-{fw_version}" + + +def _human_size(num_bytes: int) -> str: + size = float(num_bytes) + for unit in ("B", "KB", "MB", "GB"): + if size < 1024 or unit == "GB": + return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}" + size /= 1024 + + +def _short_path(path: Path) -> str: + """Path relative to the cwd when that's shorter, else absolute.""" + try: + rel = os.path.relpath(path) + except ValueError: + # Windows: path and cwd on different drives have no relative form. + return str(path) + return rel if not rel.startswith("..") else str(path) + + +def download_file(url: str, dest: Path, *, retries: int = 3) -> int: + """Download ``url`` to ``dest``; return the number of bytes written. + + Retries transient failures (connection errors and HTTP 429/5xx) with + exponential backoff - GitHub's CDN occasionally 502s under concurrent + downloads, and a sporadic failure shouldn't abort the whole fetch. + """ + for attempt in range(retries + 1): + try: + with urllib.request.urlopen(url) as resp: + status = getattr(resp, "status", 200) + if status != 200: + raise click.ClickException(f"HTTP {status} while downloading {url}") + data = resp.read() + dest.write_bytes(data) + return len(data) + except urllib.error.HTTPError as exc: + if exc.code not in _RETRY_STATUS or attempt == retries: + raise click.ClickException( + f"HTTP error while downloading {url}: {exc}" + ) from exc + reason = f"HTTP {exc.code}" + except urllib.error.URLError as exc: + if attempt == retries: + raise click.ClickException( + f"Network error while downloading {url}: {exc}" + ) from exc + reason = str(exc.reason) + delay = 0.5 * (2**attempt) + click.echo( + f" [retry] {url.rsplit('/', 1)[-1]} ({reason}); retrying in {delay:.1f}s", + err=True, + ) + time.sleep(delay) + raise click.ClickException( # pragma: no cover - loop always returns/raises + f"Failed to download {url} after {retries} retries." + ) + + +def _github_get(url: str): + """Unauthenticated GitHub API GET (60 req/hour is plenty for a CLI).""" + request = urllib.request.Request( + url, + headers={"Accept": "application/vnd.github+json", "User-Agent": "dotbot"}, + ) + try: + with urllib.request.urlopen(request) as resp: + return json.load(resp) + except (urllib.error.HTTPError, urllib.error.URLError) as exc: + raise click.ClickException(f"GitHub API request failed ({url}): {exc}") from exc + + +def resolve_release(source: str, fw_version: str) -> dict: + """Return the GitHub release JSON for ``source`` at ``fw_version``. + + ``fw_version="latest"`` resolves the newest release via ``/releases`` + (prereleases included, unlike ``/releases/latest`` which skips them - the + current newest, e.g. swarmit 0.8.0rc2, is a prerelease). + """ + if source not in RELEASE_SOURCES: + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + repo = RELEASE_SOURCES[source] + if fw_version == "latest": + releases = _github_get(f"{GITHUB_API}/{repo}/releases?per_page=1") + if not releases: + raise click.ClickException( + f"No releases found for {repo}; pass an explicit -f ." + ) + return releases[0] + return _github_get(f"{GITHUB_API}/{repo}/releases/tags/{fw_version}") + + +def resolve_latest_version(source: str = "swarmit") -> str: + """The newest release tag for ``source`` (prereleases included).""" + return resolve_release(source, "latest")["tag_name"] + + +def pinned_version(source: str) -> str: + """The exact firmware release this pydotbot pins for ``source``. + + swarmit is a Python dependency whose firmware release is tagged identically + to the package, so its pin is read from the installed package. DotBot-firmware + is not a Python package, so its pin is the declared ``DOTBOT_FIRMWARE_VERSION``. + `dotbot fw fetch` (no -f) resolves to these; pass -f to override. + """ + if source == "swarmit": + from importlib.metadata import PackageNotFoundError + from importlib.metadata import version as _pkg_version + + try: + return _pkg_version("swarmit") + except PackageNotFoundError as exc: + raise click.ClickException( + "Cannot infer the swarmit firmware version: the swarmit package " + "is not installed. Pass -f explicitly." + ) from exc + if source == "dotbot-firmware": + return DOTBOT_FIRMWARE_VERSION + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + + +def fetch_assets( + source: str, fw_version: str, bin_dir: Path, local_root: Path | None = None +) -> Path: + """Fetch one source's firmware into ``bin_dir/-/``. + + For a released version, downloads every ``.hex``/``.bin`` asset the GitHub + release publishes (so it adapts to whatever the release ships, no hardcoded + asset list) and writes a ``manifest.json`` with provenance. For + ``fw_version="local"``, symlinks/copies from a local build tree into + ``-local/``. Used by `dotbot fw fetch` and the auto-fetch hook in + `flash_role`. + """ + if source not in RELEASE_SOURCES: + raise click.ClickException( + f"Unknown firmware source '{source}'. Known: {', '.join(RELEASE_SOURCES)}." + ) + if fw_version == "local" and not local_root: + raise click.ClickException("--local-root is required when --fw-version=local.") + if fw_version != "local" and local_root: + click.echo( + "[WARN] --local-root ignored when --fw-version is not 'local'.", + err=True, + ) + + if fw_version == "local": + return _link_local_assets(source, local_root, bin_dir) + + release = resolve_release(source, fw_version) + tag = release["tag_name"] + out_dir = resolve_fw_root(bin_dir, source, tag) + out_dir.mkdir(parents=True, exist_ok=True) + assets = [ + a for a in release.get("assets", []) if a["name"].endswith((".hex", ".bin")) + ] + if not assets: + raise click.ClickException( + f"{source} release {tag} publishes no .hex/.bin assets." + ) + click.echo( + f"Fetching {source} {tag} ({len(assets)} files) into {_short_path(out_dir)}" + ) + names: list[str] = [] + errors: list[str] = [] + # Downloads are I/O-bound, so a small thread pool overlaps them (urllib + # releases the GIL during the network read). Sources stay sequential. + # Kept modest: GitHub's asset CDN starts 502ing under heavier fan-out + # (download_file retries transient failures regardless). + with ThreadPoolExecutor(max_workers=4) as pool: + future_to_name = { + pool.submit( + download_file, asset["browser_download_url"], out_dir / asset["name"] + ): asset["name"] + for asset in assets + } + for future in as_completed(future_to_name): + name = future_to_name[future] + try: + size = future.result() + except click.ClickException as exc: + errors.append(f"{name}: {exc.format_message()}") + continue + names.append(name) + click.echo(f" {name} ({_human_size(size)})") + if errors: + raise click.ClickException( + f"{len(errors)} asset(s) failed to download:\n " + "\n ".join(errors) + ) + _write_manifest(out_dir, source, tag, RELEASE_SOURCES[source], names) + return out_dir + + +def _write_manifest( + out_dir: Path, source: str, version: str, repo: str, files: list[str] +) -> None: + """Record provenance next to the binaries (a cheap audit trail).""" + from dotbot import pydotbot_version + + manifest = { + "source": source, + "version": version, + "repo": repo, + "url": f"https://github.com/{repo}/releases/tag/{version}", + "pydotbot": pydotbot_version(), + "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "files": sorted(files), + } + (out_dir / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") + + +def _link_local_assets(source: str, local_root: Path, bin_dir: Path) -> Path: + """Symlink/copy a local build tree into ``bin_dir/-local/``.""" + if source != "swarmit": + raise click.ClickException( + f"--fw-version local is only wired for source 'swarmit' so far, " + f"not '{source}'." + ) + local_root = local_root.expanduser().resolve() + out_dir = resolve_fw_root(bin_dir, source, "local") + out_dir.mkdir(parents=True, exist_ok=True) + mapping = { + "bootloader-dotbot-v3.hex": local_root + / "device/bootloader/Output/dotbot-v3/Debug/Exe/bootloader-dotbot-v3.hex", + "netcore-nrf5340-net.hex": local_root + / "device/network_core/Output/nrf5340-net/Debug/Exe/netcore-nrf5340-net.hex", + "03app_gateway_app-nrf5340-app.hex": local_root + / "mari/firmware/app/03app_gateway_app/Output/nrf5340-app/Debug/Exe/03app_gateway_app-nrf5340-app.hex", + "03app_gateway_net-nrf5340-net.hex": local_root + / "mari/firmware/app/03app_gateway_net/Output/nrf5340-net/Debug/Exe/03app_gateway_net-nrf5340-net.hex", + } + missing = [name for name, src in mapping.items() if not src.exists()] + if missing: + raise click.ClickException( + f"Missing local build artifacts: {', '.join(missing)}" + ) + for name, src in mapping.items(): + dest = out_dir / name + if dest.exists() or dest.is_symlink(): + dest.unlink() + try: + os.symlink(src, dest) + click.echo(f"[LINK] {dest} -> {src}") + except OSError: + shutil.copy2(src, dest) + click.echo(f"[COPY] {dest} <- {src}") + return out_dir diff --git a/dotbot/firmware/flash.py b/dotbot/firmware/flash.py index 389e89cf..943592dc 100644 --- a/dotbot/firmware/flash.py +++ b/dotbot/firmware/flash.py @@ -1,26 +1,23 @@ """DotBot firmware flashing + provisioning engine (no CLI). -The hardware-facing engine behind `dotbot device` and `dotbot fw fetch`: -fetch release assets, flash a role's system firmware + config page -(`flash_role`), flash a single app image (`flash_app_image`), flash the -debug-chip programmer (`flash_programmer`), and read back provisioning -state (`read_config_report`). The Click surface lives in -`dotbot/cli/device.py` and `dotbot/cli/fw.py`; this module is pure -library code. +The hardware-facing engine behind `dotbot device`: flash a role's system +firmware + config page (`flash_role`), flash a single app image +(`flash_app_image`), flash the debug-chip programmer (`flash_programmer`), +and read back provisioning state (`read_config_report`). Release fetching ++ the artifact cache live next door in `fetch.py`; the Click surface lives +in `dotbot/cli/device.py`. This module is pure library code. """ from __future__ import annotations import json import os -import shutil import time -import urllib.error -import urllib.request from pathlib import Path import click +from .fetch import fetch_assets, resolve_fw_root from .nrf import ( do_daplink, do_daplink_if, @@ -54,7 +51,6 @@ # dotbot-lh2-calibration (1-byte count + N matrices of 3x3 int32 LE). LH2_MATRIX_BYTES = 3 * 3 * 4 # 3x3 int32 matrix LH2_MAX_HOMOGRAPHIES = 16 -RELEASE_BASE_URL = "https://github.com/DotBots/swarmit/releases/download" # Application images are linked after the bootloader. APP_FLASH_BASE_ADDR = 0x00010000 # Programmer bring-up files @@ -115,31 +111,6 @@ def normalize_network_id(raw: str | None) -> tuple[int, str] | None: return value, f"{value:04X}" -def resolve_fw_root(bin_dir: Path, fw_version: str) -> Path: - return bin_dir / fw_version - - -def download_file(url: str, dest: Path) -> None: - click.echo(f"[GET ] {url}") - try: - with urllib.request.urlopen(url) as resp: - status = getattr(resp, "status", 200) - if status != 200: - raise click.ClickException(f"HTTP {status} while downloading {url}") - data = resp.read() - except urllib.error.HTTPError as exc: - raise click.ClickException( - f"HTTP error while downloading {url}: {exc}" - ) from exc - except urllib.error.URLError as exc: - raise click.ClickException( - f"Network error while downloading {url}: {exc}" - ) from exc - - dest.write_bytes(data) - click.echo(f"[OK ] wrote {dest} ({len(data)} bytes)") - - def convert_bin_to_hex(bin_path: Path, base_addr: int) -> Path: if IntelHex is None: raise click.ClickException( @@ -326,93 +297,6 @@ def manifest_matches( ) -def fetch_assets( - fw_version: str, bin_dir: Path, local_root: Path | None = None -) -> Path: - """Download (or symlink, for --fw-version=local) the testbed firmware - assets into ``bin_dir//`` and return that directory. - - The single source of truth for "get the system firmware". Used by - `dotbot fw fetch` and by the auto-fetch hook in `flash_role` - (fetch-if-absent). - """ - if fw_version == "local" and not local_root: - raise click.ClickException("--local-root is required when --fw-version=local.") - if fw_version != "local" and local_root: - click.echo( - "[WARN] --local-root ignored when --fw-version is not 'local'.", - err=True, - ) - - out_dir = resolve_fw_root(bin_dir, fw_version) - out_dir.mkdir(parents=True, exist_ok=True) - click.echo(f"[INFO] target dir: {out_dir.resolve()}") - - if fw_version == "local": - local_root = local_root.expanduser().resolve() - mapping = { - "bootloader-dotbot-v3.hex": local_root - / "device/bootloader/Output/dotbot-v3/Debug/Exe/bootloader-dotbot-v3.hex", - "netcore-nrf5340-net.hex": local_root - / "device/network_core/Output/nrf5340-net/Debug/Exe/netcore-nrf5340-net.hex", - "03app_gateway_app-nrf5340-app.hex": local_root - / "mari/firmware/app/03app_gateway_app/Output/nrf5340-app/Debug/Exe/03app_gateway_app-nrf5340-app.hex", - "03app_gateway_net-nrf5340-net.hex": local_root - / "mari/firmware/app/03app_gateway_net/Output/nrf5340-net/Debug/Exe/03app_gateway_net-nrf5340-net.hex", - } - - missing = [name for name, src in mapping.items() if not src.exists()] - if missing: - missing_list = ", ".join(missing) - raise click.ClickException(f"Missing local build artifacts: {missing_list}") - - for name, src in mapping.items(): - dest = out_dir / name - if dest.exists() or dest.is_symlink(): - dest.unlink() - try: - os.symlink(src, dest) - click.echo(f"[LINK] {dest} -> {src}") - except OSError: - shutil.copy2(src, dest) - click.echo(f"[COPY] {dest} <- {src}") - return out_dir - - assets = [ - "bootloader-dotbot-v3.hex", - "netcore-nrf5340-net.hex", - "03app_gateway_app-nrf5340-app.hex", - "03app_gateway_net-nrf5340-net.hex", - ] - # Optional sample sandbox apps. These are built from DotBot-firmware's - # apps-sandbox/ and aren't guaranteed to be on every swarmit release, so - # a 404 here is expected, not fatal — the four system images above are - # all that provisioning (flash-swarmit-sandbox / flash-mari-gateway) needs. - example_bins = [ - "dotbot-dotbot-v3.bin", - "spin-dotbot-v3.bin", - "rgbled-dotbot-v3.bin", - "move-dotbot-v3.bin", - "motors-dotbot-v3.bin", - ] - for name in assets: - url = f"{RELEASE_BASE_URL}/{fw_version}/{name}" - dest = out_dir / name - download_file(url, dest) - for name in example_bins: - url = f"{RELEASE_BASE_URL}/{fw_version}/{name}" - dest = out_dir / name - try: - download_file(url, dest) - except click.ClickException as exc: - click.echo( - f"[skip] optional sample app {name} not in release " - f"{fw_version} ({exc.format_message()})", - err=True, - ) - return out_dir - - def flash_role( role: str, *, @@ -480,14 +364,14 @@ def flash_role( calibration_hex = (bytes([count]) + matrices).hex() click.echo(f"[INFO] calibration: {count} matrices from {calibration_path}") - fw_root = resolve_fw_root(bin_dir, fw_version) + fw_root = resolve_fw_root(bin_dir, "swarmit", fw_version) # Auto-fetch: if the role's images aren't already present, pull the - # release into bin_dir// before flashing (npm-style). + # swarmit release into bin_dir/swarmit-/ before flashing. pre_app = fw_root / assets["app"] pre_net = fw_root / assets["net"] if fw_version != "local" and not (pre_app.exists() and pre_net.exists()): click.echo(f"[INFO] firmware {fw_version} not found in {fw_root}; fetching...") - fetch_assets(fw_version, bin_dir) + fetch_assets("swarmit", fw_version, bin_dir) if not fw_root.exists(): raise click.ClickException(f"Firmware root not found: {fw_root}") diff --git a/dotbot/tests/test_cli_helpers.py b/dotbot/tests/test_cli_helpers.py index fe0968f4..c7f271eb 100644 --- a/dotbot/tests/test_cli_helpers.py +++ b/dotbot/tests/test_cli_helpers.py @@ -87,8 +87,7 @@ def test_config_show_skips_none_values(runner, tmp_path): cfg = _write(tmp_path, 'swarm_id = "0001"\n') result = runner.invoke(cli, ["-c", str(cfg), "config", "show"]) assert result.exit_code == 0, result.output - # `artifacts_dir`/`log_level` are unset (None) and must not appear. - assert "artifacts_dir" not in result.output + # `log_level` is unset (None) and must not appear. assert "log_level" not in result.output diff --git a/dotbot/tests/test_device.py b/dotbot/tests/test_device.py index 08eda116..551176b2 100644 --- a/dotbot/tests/test_device.py +++ b/dotbot/tests/test_device.py @@ -10,10 +10,14 @@ board), and the friendly nrfjprog-missing error. """ +from pathlib import Path + import click import pytest from click.testing import CliRunner +import dotbot.firmware.fetch as fetch +import dotbot.firmware.flash as flash from dotbot.cli.device import _looks_like_path from dotbot.cli.device import cmd as device_cmd @@ -62,18 +66,14 @@ def test_flash_mari_gateway_rejects_calibration(runner): assert bad.exit_code != 0 -def test_flash_swarmit_sandbox_requires_swarm_id_and_version(runner): - """--swarm-id and -f are both required for flash-swarmit-sandbox.""" - assert ( - runner.invoke(device_cmd, ["flash-swarmit-sandbox", "-f", "0.8.0rc1"]).exit_code - != 0 - ) - assert ( - runner.invoke( - device_cmd, ["flash-swarmit-sandbox", "--swarm-id", "1234"] - ).exit_code - != 0 - ) +def test_flash_swarmit_sandbox_requires_swarm_id(runner): + """flash-swarmit-sandbox needs a swarm id (flag or config); -f is now + optional and defaults to the latest release (so no network in this test: + swarm_id is checked before the version resolves).""" + with runner.isolated_filesystem(): + result = runner.invoke(device_cmd, ["flash-swarmit-sandbox", "-f", "0.8.0rc1"]) + assert result.exit_code != 0 + assert "no swarm id" in result.output def test_flash_mari_gateway_help_disambiguates_from_bridge(runner): @@ -102,6 +102,24 @@ def fake_flash_role(role, **kw): assert calls["kw"]["sn_starting_digits"] == "77" +def test_flash_swarmit_sandbox_defaults_to_pinned_version( + runner, _no_nrfjprog_gate, monkeypatch +): + """With no -f, the role flash uses the pinned swarmit version (matching + `fw fetch`), not the latest release - and resolves it without the network.""" + + calls = {} + monkeypatch.setattr( + "dotbot.firmware.flash.flash_role", + lambda role, **kw: calls.update(role=role, kw=kw), + ) + result = runner.invoke( + device_cmd, ["flash-swarmit-sandbox", "--swarm-id", "0100", "-s", "77"] + ) + assert result.exit_code == 0, result.output + assert calls["kw"]["fw_version"] == fetch.pinned_version("swarmit") + + def test_flash_mari_gateway_calls_engine_with_gateway_role( runner, _no_nrfjprog_gate, monkeypatch ): @@ -332,41 +350,190 @@ def test_create_config_hex_appends_calibration(tmp_path): def test_intelhex_is_a_core_dependency(): """intelhex was folded into core deps (the [provision] extra is gone), so config-hex building works on a default `pip install pydotbot`.""" - import dotbot.firmware.flash as flash assert flash.IntelHex is not None -def test_fetch_assets_skips_missing_optional_examples(tmp_path, monkeypatch): - """A 404 on an optional sample .bin must NOT abort the fetch — the four - required system images still complete (so provisioning's auto-fetch works - even when the sample apps aren't on the release).""" - import dotbot.firmware.flash as flash +def test_fetch_assets_downloads_release_into_source_version_dir(tmp_path, monkeypatch): + """fetch_assets pulls every .hex/.bin the release lists into + -/, skips .elf/.map, and writes a manifest.""" + import json as _json - downloaded = [] + fake_release = { + "tag_name": "0.8.0rc2", + "assets": [ + {"name": "bootloader-dotbot-v3.hex", "browser_download_url": "u1"}, + {"name": "netcore-nrf5340-net.hex", "browser_download_url": "u2"}, + {"name": "bootloader-dotbot-v3.elf", "browser_download_url": "u3"}, + ], + } + monkeypatch.setattr(fetch, "resolve_release", lambda source, version: fake_release) def fake_download(url, dest): - name = url.rsplit("/", 1)[-1] - if name.endswith(".hex"): # the 4 required system images - dest.write_bytes(b"\x00") - downloaded.append(name) - else: # optional sample .bin → simulate a release 404 - raise click.ClickException(f"HTTP Error 404: {name}") - - monkeypatch.setattr(flash, "download_file", fake_download) - out = flash.fetch_assets("0.8.0rc1", tmp_path) # must not raise + dest.write_bytes(b"\x00") + return 1 + + monkeypatch.setattr(fetch, "download_file", fake_download) + out = fetch.fetch_assets("swarmit", "latest", tmp_path) + assert out == tmp_path / "swarmit-0.8.0rc2" assert (out / "bootloader-dotbot-v3.hex").exists() assert (out / "netcore-nrf5340-net.hex").exists() - assert sum(n.endswith(".hex") for n in downloaded) == 4 + assert not (out / "bootloader-dotbot-v3.elf").exists() # .elf skipped + manifest = _json.loads((out / "manifest.json").read_text()) + assert manifest["source"] == "swarmit" + assert manifest["version"] == "0.8.0rc2" + assert "bootloader-dotbot-v3.hex" in manifest["files"] + assert manifest["pydotbot"] # provenance: which pydotbot fetched this -def test_fetch_assets_still_fails_on_missing_system_image(tmp_path, monkeypatch): - """A 404 on a REQUIRED system .hex stays fatal (bad version tag).""" - import dotbot.firmware.flash as flash +def test_fetch_assets_unknown_source_errors(tmp_path): + """An unknown source is a clear error, not a KeyError.""" - def fake_download(url, dest): - raise click.ClickException("HTTP Error 404") + with pytest.raises(click.ClickException): + fetch.fetch_assets("not-a-source", "latest", tmp_path) + + +def test_resolve_latest_version_returns_newest_tag(monkeypatch): + """Returns the first (newest, prereleases included) tag from the API.""" + import io + import json + + payload = json.dumps([{"tag_name": "0.8.0rc2"}, {"tag_name": "0.8.0rc1"}]).encode() + monkeypatch.setattr( + fetch.urllib.request, "urlopen", lambda req: io.BytesIO(payload) + ) + assert fetch.resolve_latest_version() == "0.8.0rc2" + + +def test_resolve_latest_version_no_releases_errors(monkeypatch): + """An empty release list is a clear error, not an IndexError.""" + import io + + monkeypatch.setattr(fetch.urllib.request, "urlopen", lambda req: io.BytesIO(b"[]")) + with pytest.raises(click.ClickException): + fetch.resolve_latest_version() + + +def test_resolve_latest_version_network_error_errors(monkeypatch): + """A network failure surfaces as a friendly ClickException.""" + + def boom(req): + raise fetch.urllib.error.URLError("offline") + + monkeypatch.setattr(fetch.urllib.request, "urlopen", boom) + with pytest.raises(click.ClickException): + fetch.resolve_latest_version() + + +def test_download_file_retries_transient_5xx(tmp_path, monkeypatch): + """A sporadic 502 (GitHub's CDN under concurrent load) is retried, then + succeeds - one bad gateway shouldn't abort the whole fetch.""" + import io + + calls = {"n": 0} + + def flaky_urlopen(url): + calls["n"] += 1 + if calls["n"] == 1: + raise fetch.urllib.error.HTTPError(url, 502, "Bad Gateway", {}, None) + return io.BytesIO(b"\xde\xad") + + monkeypatch.setattr(fetch.urllib.request, "urlopen", flaky_urlopen) + monkeypatch.setattr(fetch.time, "sleep", lambda _delay: None) # skip real backoff + + dest = tmp_path / "spin-dotbot-v3.hex" + size = fetch.download_file("http://x/spin-dotbot-v3.hex", dest, retries=3) + assert size == 2 + assert dest.read_bytes() == b"\xde\xad" + assert calls["n"] == 2 # one retry + + +def test_download_file_gives_up_on_non_transient(tmp_path, monkeypatch): + """A 404 is not transient - it surfaces immediately, with no backoff.""" + + def not_found(url): + raise fetch.urllib.error.HTTPError(url, 404, "Not Found", {}, None) + + sleeps: list[float] = [] + monkeypatch.setattr(fetch.urllib.request, "urlopen", not_found) + monkeypatch.setattr(fetch.time, "sleep", lambda d: sleeps.append(d)) + + with pytest.raises(click.ClickException): + fetch.download_file("http://x/missing.hex", tmp_path / "missing.hex", retries=3) + assert sleeps == [] # never retried + + +def test_short_path_falls_back_to_absolute_across_drives(monkeypatch): + """On Windows os.path.relpath raises ValueError when the path and cwd are + on different drives (C: vs D:); _short_path must return the absolute path, + not crash.""" + + def boom(_p): + raise ValueError("path is on mount 'C:', start on mount 'D:'") + + monkeypatch.setattr(fetch.os.path, "relpath", boom) + p = Path("/x/swarmit-1.2.3") + assert fetch._short_path(p) == str(p) + + +def test_pinned_version_dotbot_firmware_is_declared(): + """DotBot-firmware (not a Python dep) pins to the declared constant.""" + + assert fetch.pinned_version("dotbot-firmware") == fetch.DOTBOT_FIRMWARE_VERSION + + +def test_pinned_version_swarmit_from_installed_package(): + """swarmit's firmware version is inferred from the installed package.""" + import importlib.metadata as md + + assert fetch.pinned_version("swarmit") == md.version("swarmit") + + +def test_pinned_version_unknown_source_errors(): + """An unknown source is a clear error, not a KeyError.""" - monkeypatch.setattr(flash, "download_file", fake_download) with pytest.raises(click.ClickException): - flash.fetch_assets("0.0.0-nope", tmp_path) + fetch.pinned_version("not-a-source") + + +def test_fetch_no_args_resolves_pinned_versions(monkeypatch): + """`dotbot fw fetch` with no flags fetches the pinned version per source, + not 'latest'.""" + from dotbot.cli.fw import cmd as fw_cmd + + calls: list[tuple[str, str]] = [] + monkeypatch.setattr(fetch, "pinned_version", lambda src: f"PIN-{src}") + monkeypatch.setattr( + fetch, + "fetch_assets", + lambda src, version, bin_dir, local_root=None: ( + calls.append((src, version)) or Path(f"/x/{src}-{version}") + ), + ) + res = CliRunner().invoke(fw_cmd, ["fetch"]) + assert res.exit_code == 0, res.output + assert calls == [ + ("swarmit", "PIN-swarmit"), + ("dotbot-firmware", "PIN-dotbot-firmware"), + ] + assert "latest" not in res.output # the pinned path never says "latest" + + +def test_fetch_explicit_version_overrides_pin(monkeypatch): + """-f with --source bypasses the pin and passes through verbatim.""" + from dotbot.cli.fw import cmd as fw_cmd + + calls: list[tuple[str, str]] = [] + pin_called: list[str] = [] + monkeypatch.setattr(fetch, "pinned_version", lambda src: pin_called.append(src)) + monkeypatch.setattr( + fetch, + "fetch_assets", + lambda src, version, bin_dir, local_root=None: ( + calls.append((src, version)) or Path(f"/x/{src}-{version}") + ), + ) + res = CliRunner().invoke(fw_cmd, ["fetch", "-S", "dotbot-firmware", "-f", "1.21.0"]) + assert res.exit_code == 0, res.output + assert calls == [("dotbot-firmware", "1.21.0")] + assert pin_called == [] # explicit -f never consults the pin diff --git a/pyproject.toml b/pyproject.toml index 329202dc..4c3e53f5 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -42,6 +42,7 @@ dependencies = [ "pynput >= 1.8.1", "pyserial >= 3.5", "qrkey >= 0.12.2", + "requests >= 2.31.0", "rich >= 14.0.0", "structlog >= 24.4.0", "uvicorn >= 0.32.0",