diff --git a/.claude/commands/README.md b/.claude/commands/README.md deleted file mode 100644 index 3767dac987a..00000000000 --- a/.claude/commands/README.md +++ /dev/null @@ -1,49 +0,0 @@ -# Claude Code slash commands for the mcp-server test suite - -Three AI-assisted workflows wrapping `mcp-server/run-tests.sh` and the meshtastic MCP tools. Each one has a twin in `.github/prompts/` for Copilot users. - -| Slash command | What it does | Copilot equivalent | -| --------------------- | ------------------------------------------------------------------------- | ---------------------------------------- | -| `/test [args]` | Runs the test suite (auto-detects hardware) and interprets failures | `.github/prompts/mcp-test.prompt.md` | -| `/diagnose [role]` | Read-only device health report via the meshtastic MCP tools | `.github/prompts/mcp-diagnose.prompt.md` | -| `/repro [n=5]` | Re-runs one test N times, diffs firmware logs between passes and failures | `.github/prompts/mcp-repro.prompt.md` | - -## Why two surfaces - -The Claude Code commands and Copilot prompts cover the same three workflows but each speaks its host's idiom: - -- **Claude Code** (`/test`) uses `$ARGUMENTS` for pass-through, has direct access to Bash + all MCP tools registered in the user's settings, and runs in the terminal context. -- **Copilot** (`/mcp-test`) runs in VS Code's agent mode; it has terminal + MCP access too but typically asks the operator to confirm inputs interactively. - -A contributor using either IDE gets equivalent assistance. Keep the two in sync when behavior changes — the diff of intent should be minimal. - -## House rules - -- **No destructive writes without explicit operator approval.** Skills that could reflash, factory-reset, or reboot a device must describe the action and stop — the operator authorizes. -- **Interpret failures, don't just echo them.** The skill body should pull firmware log lines from `mcp-server/tests/report.html` (the `Meshtastic debug` section, attached by `tests/conftest.py::pytest_runtest_makereport`) and classify the failure. -- **Keep MCP tool calls sequential per port.** SerialInterface holds an exclusive port lock; two parallel tool calls on the same port deadlock. -- **Never speculate about root cause.** If the evidence doesn't support a classification, say "unknown" and list what you'd need to disambiguate. - -## Adding a new command - -1. Write the Claude Code version at `.claude/commands/.md` with YAML frontmatter: - - ```yaml - --- - description: one-line purpose (used for auto-invocation by the model) - argument-hint: [optional-hint] - --- - ``` - -2. Write the Copilot equivalent at `.github/prompts/mcp-.prompt.md` with: - - ```yaml - --- - mode: agent - description: ... - --- - ``` - -3. Add the row to the table above. Cross-link in both bodies. - -4. Smoke-test on Claude Code first (`/` should appear in autocomplete), then in VS Code Copilot (`/mcp-` in Chat). diff --git a/.claude/commands/diagnose.md b/.claude/commands/diagnose.md deleted file mode 100644 index d664f631294..00000000000 --- a/.claude/commands/diagnose.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -description: Produce a device health report using the meshtastic MCP tools (device_info, list_nodes, get_config, short serial log capture) -argument-hint: [role=all|nrf52|esp32s3|] ---- - -# `/diagnose` — device health report - -Call the meshtastic MCP tool bundle and format a structured health report for one or all detected devices. Zero guesswork for the operator. - -## What to do - -1. **Enumerate hardware.** Call `mcp__meshtastic__list_devices(include_unknown=True)`. For each entry where `likely_meshtastic=True`, capture `port`, `vid`, `pid`, `description`. - -2. **Filter by `$ARGUMENTS`**: - - No args, `all` → every likely-meshtastic device. - - `nrf52` → only devices with `vid == 0x239a`. - - `esp32s3` → only devices with `vid == 0x303a` or `vid == 0x10c4`. - - A `/dev/cu.*` path → only that one port. - - Anything else → treat as a substring match against the `port` string. - -3. **For each selected device, in sequence (NOT parallel — SerialInterface holds an exclusive port lock):** - - `mcp__meshtastic__device_info(port=

)` — captures `my_node_num`, `long_name`, `short_name`, `firmware_version`, `hw_model`, `region`, `num_nodes`, `primary_channel`. - - `mcp__meshtastic__list_nodes(port=

)` — count of peers, which ones have `publicKey` set, SNR/RSSI distribution. - - `mcp__meshtastic__get_config(section="lora", port=

)` — region, preset, channel_num, tx_power, hop_limit. - - Optionally, if the device seems unhappy (fails to connect, `num_nodes==1` when ≥2 are plugged in, missing firmware*version), open a short firmware log window: `mcp__meshtastic__serial_open(port=

, env=)`, wait 3s, `serial_read(session_id=, max_lines=100)`, `serial_close(session_id=)`. The env should be inferred from the VID map in `mcp-server/run-tests.sh` (nrf52 → rak4631, esp32s3 → heltec-v3) unless `MESHTASTIC_MCP_ENV*` is set. - -4. **Hub health** (call once, not per-device): `mcp__meshtastic__uhubctl_list()` — enumerates every USB hub the host can see. Note which hubs advertise `ppps=true` and which hub hosts each Meshtastic device (cross-reference by VID). Flag it in the report if: - - No hub advertises PPPS → `tests/recovery/` can't run on this setup; hard-recovery via `uhubctl_cycle` isn't available. - - A Meshtastic device is on a non-PPPS hub → note it; operator may want to move the device to a PPPS hub to unlock auto-recovery. - - `uhubctl_list` raises `ConfigError: uhubctl not found` → just say `uhubctl not installed` in the report; don't treat as a fault. - -5. **Render per-device report** as: - - ```text - [nrf52 @ /dev/cu.usbmodem1101] fw=2.7.23.bce2825, hw=RAK4631 - owner : Meshtastic 40eb / 40eb - region/band : US, channel 88, LONG_FAST - tx_power : 30 dBm, hop_limit=3 - peers : 1 (esp32s3 0x433c2428, pubkey ✓, SNR 6.0 / RSSI -24 dBm) - primary ch : McpTest - hub : 1-1.3 port 2 (PPPS, uhubctl-controllable) - firmware : no panics in last 3s; NodeInfoModule emitted 2 broadcasts - ``` - - Keep it scannable. If a field is missing or abnormal (no pubkey for a known peer, region=UNSET, num_nodes inconsistent with the hub, device on non-PPPS hub), flag it inline with a short `⚠︎ `. - -6. **Cross-device correlation** (only when >1 device is inspected): - - Do both sides see each other in `nodesByNum`? If one does and the other doesn't, that's asymmetric NodeInfo — flag it. - - Do the LoRa configs match? (region, channel_num, modem_preset should all agree; mismatch = no mesh) - - Do the primary channel NAMES match? Mismatch = different PSK = no decode. - -7. **Recorder slice (cheap, always available).** The mcp-server runs an autouse log recorder that's been collecting from every connected device. Pull two short slices to surface anything weird that's already happened: - - `mcp__meshtastic__logs_window(start="-2m", level="WARN|ERROR|CRIT", max_lines=20)` — recent firmware errors. If empty, say "no recent errors"; don't manufacture concern. - - `mcp__meshtastic__telemetry_timeline(window="1h", field="free_heap", max_points=60)` — heap trend. If `slope_per_min < -50`, flag it and recommend `/leakhunt window=6h` for a deeper read; otherwise just note the current free heap. - - If `recorder_status` shows `running:false` or `files.telemetry.last_ts` is null, note "recorder has no telemetry yet — enable `set_debug_log_api(True)` to populate" and skip this step gracefully. - -8. **Suggest next actions only for specific, recognisable failure modes**: - - Stale PKI pubkey one-way → "run `/test tests/mesh/test_direct_with_ack.py` — the retry + nodeinfo-ping heals this in the test path." - - Region mismatch → "re-bake one side via `./mcp-server/run-tests.sh --force-bake`." - - Device unreachable, reachable via DFU → `touch_1200bps(port=...)` + `pio_flash`. If not even DFU responds AND the device is on a PPPS hub, escalate to `uhubctl_cycle(role=..., confirm=True)`. - - CP2102-wedged-driver on macOS → see the note in `run-tests.sh`. - - Heap slope strongly negative → "run `/leakhunt window=6h` for a full timeline + classification." - -## What NOT to do - -- No writes. No `set_config`, no `reboot`, no `factory_reset`. This is a read-only diagnostic skill — if the operator wants to change state, they'll ask explicitly. -- No `flash` / `erase_and_flash`. Those are separate escalations. -- No holding SerialInterface across tool calls — open, query, close; next device. The port lock is exclusive. diff --git a/.claude/commands/leakhunt.md b/.claude/commands/leakhunt.md deleted file mode 100644 index ef90b133e24..00000000000 --- a/.claude/commands/leakhunt.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -description: Hunt for memory leaks (and other slow degradations) by reading the persistent recorder's heap timeline + log slice over a window -argument-hint: [window=1h] [field=free_heap] [variant=local] ---- - - - -# `/leakhunt` — read the recorder, classify a memory leak - -Use the always-on recorder (`mcp-server/.mtlog/`) to read a heap timeline plus the matching log slice and produce a one-page verdict: **steady / slow leak / fragmentation / OOM-imminent**. No firmware changes, no special build flags — the LocalStats telemetry packet that the firmware already broadcasts every ~60 s carries `heap_free_bytes` and `heap_total_bytes`. - -## Two signal paths — pick the right one - -| Path | Build flag | Cadence | Per-thread attribution | Cost | -| --------------------- | ---------------- | -------------- | ---------------------- | ------------------------- | -| LocalStats packet | (default) | ~60 s | No | Free — always on | -| `[heap N]` log prefix | `-DDEBUG_HEAP=1` | every log line | Yes (Thread X leaked) | Bigger flash + log volume | - -Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_HEAP is on, the recorder synthesizes telemetry rows from log prefixes (tagged `source: debug_heap`), so a single timeline call gets whichever signal is available. **For a slow leak diagnosis, the default path is plenty** (60 s cadence over 6 h = 360 points; linear regression over that nails sub-100-byte/min slopes). **DEBUG_HEAP is for attribution** — when the slope is real and you need to know which thread is leaking. - -## What to do - -1. **Parse `$ARGUMENTS`**: optional `window` (default `1h`, accepts `30m`/`6h`/`-3d`/etc.), optional `field` (default `free_heap`; alternates: `total_heap`, `battery_level`, anything in the LocalStats variant), optional `variant` (default `local`; alternates: `device`, `environment`, `power`, `airQuality`, `health`). - -2. **Verify the recorder is alive** — call `mcp__meshtastic__recorder_status`. Check: - - `running == True` - - `files.telemetry.lines > 0` (at least one telemetry packet recorded — if zero, the device hasn't broadcast LocalStats yet OR `set_debug_log_api` has never been on; tell the operator to run `mcp__meshtastic__set_debug_log_api(enabled=True)` and wait one device-update interval) - - `files.telemetry.last_ts` within the last 5 minutes (if older, the device is silent — log that, not "leak detected") - -3. **Detect whether DEBUG_HEAP is active** — `mcp__meshtastic__logs_window(start="-2m", grep=r"\\[heap \\d+\\]", max_lines=3)`. If any line matches, the firmware has the prefix → DEBUG_HEAP is on, expect higher-cadence data and `heap_event` rows. If zero matches over the last 2 minutes, you're on the LocalStats-only path. - -4. **Pull the timeline** — `mcp__meshtastic__telemetry_timeline(window=$window, variant=$variant, field=$field, max_points=200)`. Read: - - `samples` — how many raw points contributed - - `min`, `max` — total swing - - `slope_per_min` — units per minute (linear regression over the whole window) - -5. **Pull the log context for the same window** — `mcp__meshtastic__logs_window(start="-${window}", grep="Heap status|leaked heap|freed heap|out of memory|Alloc an err|panic|abort", max_lines=200)`. These are the strings the firmware emits when something memory-related happens (`DEBUG_HEAP` builds emit `"Heap status:"` and `"leaked heap"` lines; production builds emit `"Alloc an err"` on failure and `"out of memory"` on OOM). - -6. **Pull marker events** so we know if the operator labeled phases — `mcp__meshtastic__events_window(start="-${window}", kind="mark|connection_lost|connection_established")`. If a `connection_lost` overlaps a sharp drop, that's not a leak; that's a reboot. - -6a. **(DEBUG_HEAP only) Per-thread attribution** — `mcp__meshtastic__logs_window(start="-${window}", grep="leaked heap", max_lines=200)`. Each row has a structured `heap_event` field with `{kind, thread, before, after, delta}`. Aggregate by thread: sum the `delta` over the window per thread name. The thread with the largest cumulative negative delta is your suspect. Note the count too — a thread with 50× small leaks is different from 1× big leak. - -7. **Classify** based on what the data says, NOT on what you wish it said. Use these rules in order: - - **Insufficient data** (< 5 samples): say so. Suggest a longer window or longer wait. Stop. - - **Reboot mid-window**: if any `connection_lost` event is present AND `free_heap` jumped UP at that timestamp, the device rebooted. Note it; pre-reboot trend may be a leak but you only have part of the curve. - - **OOM-imminent**: any `Alloc an err=` or `out of memory` line in the log slice. This trumps everything; flag urgently. - - **Slow leak**: `slope_per_min < -50` AND `max - min > 1000` AND no reboot. The heap is monotonically (or near-monotonically) declining. Estimate time-to-zero: `min / -slope_per_min` minutes. Surface it. - - **Fragmentation suspect**: `slope_per_min` close to zero (|x| < 50) BUT min trends down across the window AND the log slice shows `Alloc an err` warnings WITHOUT total OOM. Means free total is OK but largest contiguous block is shrinking. Recommend a `DEBUG_HEAP` build to confirm. - - **Steady**: |slope_per_min| < 50, no error lines. Heap is fine. - - **Recovery curve**: slope is POSITIVE — heap recovered. Either a workload completed or GC fired. Note it; not a leak. - -8. **Report**: - - ```text - /leakhunt window=6h field=free_heap variant=local - ──────────────────────────────────────────────────── - recorder : running, telem last_ts 8s ago - build : DEBUG_HEAP=ON (per-line prefix detected) - samples : 14,200 over 6h (cadence ~1.5s, log-line synth) - free_heap : min 92,344 / max 124,008 / range 31,664 - slope : -82 bytes/min (negative — heap declining) - reboots : none in window - OOM events : none - error lines : 3× "Alloc an err=ESP_ERR_NO_MEM" at +4h12m, +5h08m, +5h44m - thread leaks : (DEBUG_HEAP) MeshPacket -3,124 B over 18 events - Router -1,408 B over 4 events - others -240 B - verdict : SLOW LEAK — primary suspect MeshPacket thread - est. time-to-OOM: ~1,127 min (~18.8 h) at current slope - evidence : (3 log line citations with uptimes) - ``` - - Then: **what to do next.** - - SLOW LEAK, **DEBUG_HEAP off** → recommend rebuilding with the flag and re-running this skill. Concrete one-liner the operator can copy: - ```text - mcp__meshtastic__build(env="", build_flags={"DEBUG_HEAP": 1}) - mcp__meshtastic__pio_flash(env="", port="", confirm=True) - ``` - After flash, set debug_log_api back on and wait one window; re-run `/leakhunt`. - - SLOW LEAK, **DEBUG_HEAP on** → cite the top-leaking thread name from step 6a. Point at the corresponding source file (`grep -rn "ThreadName(\"\")" src/`); the operator decides what to fix. - - FRAGMENTATION SUSPECT → propose pre-allocating any per-packet buffers; or rebuilding with `CONFIG_HEAP_TASK_TRACKING=y` on ESP32 to see who's holding the largest blocks. - - OOM-IMMINENT → flag for immediate attention; don't wait for the next telemetry interval. - - STEADY → say so; stop. Don't invent problems. - -## What NOT to do - -- Don't assume a leak from a single dip. LocalStats fires every ~60 s and the firmware naturally allocates+frees on each broadcast cycle; one packet sees the trough. Look at the slope, not the deltas. -- Don't recommend code changes. This skill diagnoses; the operator decides what to fix. -- Don't enable `set_debug_log_api` automatically — if it's off, telemetry isn't reaching pubsub anyway, and the recorder will be empty. Tell the operator to flip it on and wait, then re-run. -- Don't run heavy workloads to "trigger the leak." The recorder is passive; we read what's there. - -## Companion: `mark_event` for stress runs - -If the operator wants to test under stimulus (e.g. blast 50 broadcasts and see what the heap does), they can frame the experiment with markers: - -```text -mark_event("burst-start") -… run the workload … -mark_event("burst-end") -/leakhunt window=15m -``` - -The markers land in both `events.jsonl` and `logs.jsonl`, so the report can show "free_heap dipped 8 KB during the burst window, recovered to baseline within 2 LocalStats cycles" → not a leak. diff --git a/.claude/commands/repro.md b/.claude/commands/repro.md deleted file mode 100644 index 84513e45b18..00000000000 --- a/.claude/commands/repro.md +++ /dev/null @@ -1,70 +0,0 @@ ---- -description: Re-run a specific test N times in isolation to triage flakes, diff firmware logs between passes and failures -argument-hint: [count=5] ---- - - - -# `/repro` — flakiness triage for one test - -Re-run a single pytest node ID N times in isolation, track pass rate, and surface what's _different_ in the firmware logs between the passing attempts and the failing ones. Turns "it's flaky, I guess" into "it fails when X, passes when Y." - -## What to do - -1. **Parse `$ARGUMENTS`**: first token is the pytest node id (e.g. `tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[nrf52->esp32s3]`); second token is an integer count (default `5`, cap at `20`). If the first token doesn't look like a test path (no `::` and no `tests/` prefix), treat the whole `$ARGUMENTS` as a `-k` filter instead. - -2. **Sanity-check the hub first** (so we're not measuring "nothing plugged in" N times): call `mcp__meshtastic__list_devices`. If the test name contains `nrf52` or `esp32s3` and the matching VID isn't present, stop and report — re-running won't help. - -3. **Loop N times**. For each iteration: - - ```bash - ./mcp-server/run-tests.sh --tb=short -p no:cacheprovider - ``` - - Capture: exit code, duration, and (on failure) the `Meshtastic debug` firmware log section from `mcp-server/tests/report.html`. `-p no:cacheprovider` suppresses pytest's `.pytest_cache` writes so iterations don't influence each other. - -4. **Track a small structured tally**: - - ```text - attempt 1: PASS (42s) - attempt 2: FAIL (128s) ← firmware log 200-line tail captured - attempt 3: PASS (39s) - attempt 4: FAIL (121s) - attempt 5: PASS (41s) - -------------------------------------- - pass rate: 3/5 (60%) | mean duration: 74s - ``` - -5. **On mixed outcomes**: diff the firmware log tails between a representative passing attempt and a representative failing attempt. Focus on: - - Error-level lines only present in failures (`PKI_UNKNOWN_PUBKEY`, `Alloc an err=`, `Skip send`, `No suitable channel`) - - Timing around the assertion event — did a broadcast go out, was there an ACK, did NAK fire? - - Device state fields that changed (nodesByNum entries, region/preset, channel_num) - - Surface the top 3 differences as a "passes when / fails when" table. Don't dump full logs — pull specific lines with uptime timestamps. - -5a. **Archive recorder slices per attempt** (no extra device interaction; the recorder runs autouse). Right after each attempt finishes, capture its `(start_ts, end_ts)` and call `mcp__meshtastic__recorder_export(start=, end=, dest_dir="mcp-server/tests/repro_artifacts//attempt_/")`. This drops a `logs.jsonl`, `telemetry.jsonl`, `packets.jsonl`, and `events.jsonl` snapshot scoped to the attempt window. Use these for cross-attempt diffs in step 5: `jq '.line' logs.jsonl` is faster than re-running the test, and the telemetry slice lets you compare heap behavior across attempts. - -6. **Classify the flake** into one of: - - **LoRa airtime collision** → pass rate improves with fewer concurrent transmitters; propose a `time.sleep` gap or retry bump in the test body. - - **PKI key staleness** → fails on first attempt, passes after self-heal; existing retry loop in `test_direct_with_ack.py` handles this. - - **NodeInfo cooldown** → `Skip send NodeInfo since we sent it <600s ago` in fail-only logs; needs `broadcast_nodeinfo_ping()` warmup. - - **Hardware-specific** (one direction fails, other passes; one device's firmware is older; driver wedged) → specific recovery pointer. For a device that's wedged past `touch_1200bps`, the next escalation is `uhubctl_cycle(role=..., confirm=True)` to hard-power-cycle its hub port (requires `uhubctl` installed). - - **Device went dark mid-run** → fails from some attempt onward, never recovers, firmware log stops arriving. Almost always hardware: a Guru crash + frozen CDC. Hard-power-cycle via `uhubctl_cycle(role=..., confirm=True)` before the next iteration; if that also fails, escalate to replug. - - **Genuinely unknown** → say so; don't invent a root cause. - -7. **Report back** with: - - Pass rate and mean duration. - - Classification + evidence (the specific log lines that support it). - - A suggested next step (re-run with specific args, open `/diagnose`, edit a specific test file, nothing). - -## Examples - -- `/repro tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[esp32s3->nrf52] 10` — runs 10 times, diffs firmware logs. -- `/repro broadcast_delivers` — no `::`, no `tests/`, so interpreted as `-k broadcast_delivers`; runs every matching test the default 5 times. -- `/repro tests/telemetry/test_device_telemetry_broadcast.py 3` — shorter run for a slow test. - -## Constraints - -- Don't exceed `count=20` per invocation — airtime and USB wear add up. If the user asks for 50, negotiate down. -- Don't rebuild firmware as part of triage; flakes that only reproduce under different firmware belong in a separate session. -- If the FIRST attempt fails AND the rest all pass, that's a classic "state leak from a prior test" → say so and suggest running with `--force-bake` or starting from a clean state rather than chasing the first failure. diff --git a/.claude/commands/test.md b/.claude/commands/test.md deleted file mode 100644 index 46a753749a3..00000000000 --- a/.claude/commands/test.md +++ /dev/null @@ -1,47 +0,0 @@ ---- -description: Run the mcp-server test suite (auto-detects devices) and interpret the results -argument-hint: [pytest-args] ---- - -# `/test` — mcp-server test runner with interpretation - -Run `mcp-server/run-tests.sh` and make sense of the output so the operator doesn't have to. - -## What to do - -1. **Invoke the wrapper.** From the firmware repo root, run: - - ```bash - ./mcp-server/run-tests.sh $ARGUMENTS - ``` - - The wrapper auto-detects connected Meshtastic devices, maps each to its PlatformIO env, exports the required `MESHTASTIC_MCP_ENV_*` env vars, and invokes pytest. If the user passed no arguments, the wrapper supplies a sensible default set (`tests/ --html=tests/report.html --self-contained-html --junitxml=tests/junit.xml -v --tb=short`). A `--report-log=tests/reportlog.jsonl` arg is always appended (unless the operator passed their own). `--assume-baked` is deliberately NOT in the defaults — `test_00_bake.py` has its own skip-if-already-baked check and runs the ~8 s verification by default. Operators can opt into the fast path with `--assume-baked`, or force a reflash with `--force-bake`. - -2. **Read the pre-flight header.** First ~6 lines print the detected hub (role → port → env). If that line reads `detected hub : (none)`, the wrapper will narrow to `tests/unit` only — say so explicitly in your summary so the operator knows hardware tiers were skipped. - -3. **On pass**: one-line summary of the form `N passed, M skipped in `. Don't enumerate the test names — the user can read those. Do mention any SKIPPED tests and name the cause: - - `"role not present on hub"` → device unplugged; operator knows to reconnect. - - `"firmware not baked with USERPREFS_UI_TEST_LOG"` → tests/ui skipped because the macro isn't in firmware yet; suggest `--force-bake`. - - `"uhubctl not installed"` → tests/recovery + peer-offline skipped; suggest `brew install uhubctl` / `apt install uhubctl`. - - `"no PPPS-capable hubs detected"` → tests/recovery skipped because the hub doesn't support per-port power; the tier will never run on that setup. - - `"opencv-python-headless is not installed"` → tests/ui auto-deselected by run-tests.sh; suggest `pip install -e 'mcp-server/.[ui]'`. - -4. **On failure**: for every FAILED test, open `mcp-server/tests/report.html` and extract the `Meshtastic debug` section for that test. pytest-html embeds the firmware log stream + device state dump there; the 200-line firmware log tail is usually enough to explain the failure. Summarise: which test, one-line assertion message, the firmware log lines that matter (things like `PKI_UNKNOWN_PUBKEY`, `Skip send NodeInfo`, `Error=`, `Guru Meditation`, `assertion failed`). For UI-tier failures also glance at `mcp-server/tests/ui_captures///transcript.md` — it records each step's frame + OCR. - -5. **Classify the failure** as one of: - - **Transient/flake**: LoRa collision, timing-sensitive assertion, first-attempt NAK + successful retry pattern. Propose `/repro ` to confirm. - - **Environmental**: device unreachable, port busy, CP2102 driver wedged. Suggest the specific recovery in escalation order: (a) replug USB, (b) `touch_1200bps(port=...)` + `pio_flash` for nRF52 DFU, (c) `uhubctl_cycle(role="nrf52", confirm=True)` when a device is fully wedged past DFU (needs `uhubctl` installed — `baked_single`'s auto-recovery hook does this once automatically). Also check `git status userPrefs.jsonc`. - - **Regression**: same assertion fails repeatedly, firmware log shows a new/unusual error. Surface the diff between expected and observed, identify the module likely responsible. - -6. **Never run destructive recovery automatically.** If a failure looks like it needs a reflash, factory*reset, `uhubctl_cycle`, or USB replug, \_describe what to do* — don't execute. The operator decides. - -## Arguments handling - -- No args → wrapper's defaults (full suite). -- `$ARGUMENTS` passed verbatim to the wrapper, which passes them to pytest. -- Common operator invocations: `/test tests/mesh`, `/test tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip`, `/test --force-bake`, `/test -k telemetry`. - -## Side-effects to mention in summary - -- The session fixture snapshots `userPrefs.jsonc` at session start and restores at teardown (plus on `atexit`). After a clean run, `git status userPrefs.jsonc` should be empty. If the wrapper's pre-flight printed a warning about a stale sidecar, call that out — means a prior session crashed. -- `mcp-server/tests/report.html` and `junit.xml` are regenerated on every run; the HTML is self-contained (shareable). diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 00000000000..9df11ff2e7b --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,17 @@ +{ + "hooks": { + "PostToolUse": [ + { + "matcher": "Write|Edit", + "hooks": [ + { + "type": "command", + "command": "f=$(tr -d '\\n' | grep -o '\"file_path\"[[:space:]]*:[[:space:]]*\"[^\"]*\"' | head -1 | sed 's/.*:[[:space:]]*\"//; s/\"$//'); [ -n \"$f\" ] && [ -f \"$f\" ] || exit 0; t=$(command -v trunk || echo \"$HOME/.cache/trunk/launcher/trunk\"); [ -x \"$t\" ] || { echo \"trunk-fmt hook: trunk not found; its launcher needs curl or wget to bootstrap the CLI (see 'Formatting & the trunk toolchain' in .github/copilot-instructions.md)\" >&2; exit 1; }; out=$(\"$t\" fmt --force \"$f\" 2>&1) || { echo \"trunk-fmt hook: trunk fmt failed on $f: $out\" >&2; exit 1; }", + "timeout": 120, + "statusMessage": "Formatting (trunk)..." + } + ] + } + ] + } +} diff --git a/.clusterfuzzlite/Dockerfile b/.clusterfuzzlite/Dockerfile index 6114417c986..86de4f5db83 100644 --- a/.clusterfuzzlite/Dockerfile +++ b/.clusterfuzzlite/Dockerfile @@ -18,7 +18,7 @@ ENV PIP_ROOT_USER_ACTION=ignore # trunk-ignore(hadolint/DL3008): apt packages are not pinned. # trunk-ignore(terrascan/AC_DOCKER_0002): apt packages are not pinned. RUN apt-get update && apt-get install --no-install-recommends -y \ - cmake git zip libgpiod-dev libbluetooth-dev libi2c-dev \ + cmake git zip libgpiod-dev libjsoncpp-dev libbluetooth-dev libi2c-dev \ libunistring-dev libmicrohttpd-dev libgnutls28-dev libgcrypt20-dev \ libusb-1.0-0-dev libssl-dev pkg-config libsqlite3-dev libsdl2-dev && \ apt-get clean && rm -rf /var/lib/apt/lists/* && \ diff --git a/.clusterfuzzlite/build.sh b/.clusterfuzzlite/build.sh index 86ab775f941..64bbbd56c62 100644 --- a/.clusterfuzzlite/build.sh +++ b/.clusterfuzzlite/build.sh @@ -31,7 +31,7 @@ cmake --install "$WORK/ulfius/$SANITIZER" --prefix /usr cd "$SRC/firmware" PLATFORMIO_EXTRA_SCRIPTS=$(echo -e "pre:.clusterfuzzlite/platformio-clusterfuzzlite-pre.py\npost:.clusterfuzzlite/platformio-clusterfuzzlite-post.py") -STATIC_LIBS=$(pkg-config --libs --static libulfius openssl libgpiod yaml-cpp bluez --silence-errors) +STATIC_LIBS=$(pkg-config --libs --static libulfius openssl libgpiod yaml-cpp jsoncpp bluez --silence-errors) export PLATFORMIO_EXTRA_SCRIPTS export STATIC_LIBS export PLATFORMIO_WORKSPACE_DIR="$WORK/pio/$SANITIZER" diff --git a/.coderabbit.yaml b/.coderabbit.yaml new file mode 100644 index 00000000000..83f59cca038 --- /dev/null +++ b/.coderabbit.yaml @@ -0,0 +1,33 @@ +# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json +# CodeRabbit config - see https://docs.coderabbit.ai/getting-started/yaml-configuration +language: en-US + +reviews: + # chill = fewer nitpicks. + profile: chill + high_level_summary: true + poem: false + auto_review: + enabled: true + # Don't burn reviews on WIP. This repo opens lots of draft PRs; review on "ready". + drafts: false + # Review PRs regardless of target branch. + base_branches: + - ".*" + # Skip Renovate dependency updates - CI gates dependencies; we review for substance, not every bump. + ignore_usernames: + - renovate + - renovate[bot] + # Stop reviewing once a PR is closed. + abort_on_close: true + + path_filters: + # Generated - don't review, just noise + token burn. + - "!src/mesh/generated/**" + - "!protobufs/**" + + path_instructions: + - path: "bin/config.d/**" + instructions: > + meshtasticd configuration files. Bundled with meshtasticd Linux/MacOS packaging. + Ensure configurations include metadata found in other configs. diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile index f546d4cfd2c..d10db534f1b 100644 --- a/.devcontainer/Dockerfile +++ b/.devcontainer/Dockerfile @@ -16,6 +16,7 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \ libssl-dev \ libulfius-dev \ libyaml-cpp-dev \ + libjsoncpp-dev \ pipx \ pkg-config \ python3 \ diff --git a/.github/actions/build-variant/action.yml b/.github/actions/build-variant/action.yml index e937966149e..8dab42f0ec6 100644 --- a/.github/actions/build-variant/action.yml +++ b/.github/actions/build-variant/action.yml @@ -76,7 +76,7 @@ runs: done - name: PlatformIO ${{ inputs.arch }} download cache - uses: actions/cache@v5 + uses: actions/cache@v6 with: path: ~/.platformio/.cache key: pio-cache-${{ inputs.arch }}-${{ hashFiles('.github/actions/**', '**.ini') }} diff --git a/.github/actions/setup-base/action.yml b/.github/actions/setup-base/action.yml index 8e461998a3a..99f9e62cfed 100644 --- a/.github/actions/setup-base/action.yml +++ b/.github/actions/setup-base/action.yml @@ -5,7 +5,7 @@ runs: using: composite steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive @@ -13,7 +13,7 @@ runs: shell: bash run: | sudo apt-get -y update --fix-missing - sudo apt-get install -y cppcheck libbluetooth-dev libgpiod-dev libyaml-cpp-dev lsb-release + sudo apt-get install -y cppcheck libbluetooth-dev libgpiod-dev libyaml-cpp-dev libjsoncpp-dev lsb-release - name: Setup Python uses: actions/setup-python@v6 diff --git a/.github/actions/setup-native/action.yml b/.github/actions/setup-native/action.yml index 05f95cd406e..3a38a4a22a9 100644 --- a/.github/actions/setup-native/action.yml +++ b/.github/actions/setup-native/action.yml @@ -11,4 +11,4 @@ runs: - name: Install libs needed for native build shell: bash run: | - sudo apt-get install -y libbluetooth-dev libgpiod-dev libyaml-cpp-dev openssl libssl-dev libulfius-dev liborcania-dev libusb-1.0-0-dev libi2c-dev libuv1-dev + sudo apt-get install -y libbluetooth-dev libgpiod-dev libyaml-cpp-dev libjsoncpp-dev openssl libssl-dev libulfius-dev liborcania-dev libusb-1.0-0-dev libi2c-dev libuv1-dev diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8573b5f4512..f0c456f4b71 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1,5 +1,23 @@ # Meshtastic Firmware - Copilot Instructions +> **TL;DR** +> +> | | | +> | -------------- | ---------------------------------------------------------------------------------------------------------------------- | +> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) | +> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | +> | Format | `trunk fmt` | +> | Mirror docs | `AGENTS.md` (short pointer for agents that don't read this file) · `CLAUDE.md` (Claude Code) | +> +> **Need this? It's here.** +> +> | | | +> | ------------------------------------------- | ---------------------------------------------------------- | +> | General helpers (clamp, UTF-8, string fmt…) | `src/meshUtils.h` | +> | Logging macros (LOG_DEBUG / INFO / WARN…) | `src/DebugConfiguration.h` | +> | New module skeleton | inherit `ProtobufModule` in `src/mesh/ProtobufModule.h` | +> | Observer / event wiring | `src/Observer.h` | + This document provides context and guidelines for AI assistants working with the Meshtastic firmware codebase. ## Project Overview @@ -37,7 +55,7 @@ MQTT provides a bridge between Meshtastic mesh networks and the internet, enabli Messages are published/subscribed using a hierarchical topic format: -``` +```text {root}/{channel_id}/{gateway_id} ``` @@ -73,15 +91,15 @@ PKI (Public Key Infrastructure) messages have special handling: ## Encryption & Key Management -Meshtastic packets on the air are typically encrypted one of two ways: the **per-channel symmetric** layer (AES-CTR with a shared PSK) for broadcasts and channel traffic, and the **per-peer PKI** layer (X25519 ECDH → AES-256-CCM) for direct messages and remote admin. A channel with a 0-byte PSK (or Ham mode, which wipes PSKs) transmits cleartext — see the size table below. Both are implemented in `src/mesh/CryptoEngine.cpp`; the send/receive dispatch lives in `src/mesh/Router.cpp`; admin authorization lives in `src/modules/AdminModule.cpp`. +Meshtastic packets on the air are typically encrypted one of two ways: the **per-channel symmetric** layer (AES-CTR with a shared PSK) for broadcasts and channel traffic, and the **per-peer PKI** layer (X25519 ECDH → AES-256-CCM) for direct messages and remote admin. A channel with a 0-byte PSK (or Ham mode, which wipes PSKs) transmits cleartext - see the size table below. Both are implemented in `src/mesh/CryptoEngine.cpp`; the send/receive dispatch lives in `src/mesh/Router.cpp`; admin authorization lives in `src/modules/AdminModule.cpp`. ### High-level model -- **Channels** are symmetric rooms: anyone with the PSK can read any message on the channel. Channel 0 is the "primary" channel and ships with the short-form default PSK on factory devices, forming the public mesh most users join. (The LoRa modem preset `LONG_FAST` lives on `config.lora.modem_preset` and is an independent field — don't conflate "channel 0 default PSK" with the modem preset name.) +- **Channels** are symmetric rooms: anyone with the PSK can read any message on the channel. Channel 0 is the "primary" channel and ships with the short-form default PSK on factory devices, forming the public mesh most users join. (The LoRa modem preset `LONG_FAST` lives on `config.lora.modem_preset` and is an independent field - don't conflate "channel 0 default PSK" with the modem preset name.) - **DMs** addressed to a single node require PKI so that other holders of the channel PSK can't read them. Outside Ham mode, Meshtastic does not fall back to channel-symmetric encryption when the destination public key is unknown. - **Remote admin** is a DM carrying an `AdminMessage`. The receiver only acts on it if the sender's public key is on its allowlist (`config.security.admin_key[0..2]`). -- **Ham mode** (`owner.is_licensed=true`, where `owner` is the local `meshtastic_User` record) disables PKI entirely and sends cleartext — FCC Part 97 prohibits encryption on amateur bands. -- **No ratchet, no session.** Every packet is encrypted from scratch — a stateless design that matches the high-loss, store-and-forward nature of LoRa. +- **Ham mode** (`owner.is_licensed=true`, where `owner` is the local `meshtastic_User` record) disables PKI entirely and sends cleartext - FCC Part 97 prohibits encryption on amateur bands. +- **No ratchet, no session.** Every packet is encrypted from scratch - a stateless design that matches the high-loss, store-and-forward nature of LoRa. ### Symmetric channel encryption (AES-CTR) @@ -104,40 +122,150 @@ Meshtastic packets on the air are typically encrypted one of two ways: the **per - **Keypair**: Curve25519 (aka X25519), 32-byte public + 32-byte private. Stored in `config.security.public_key` / `private_key`; the public half is mirrored into `owner.public_key` so it rides along in NodeInfo broadcasts and propagates through the mesh like any other identity field. - **Key generation** (`generateKeyPair`): stirs `HardwareRNG::fill()` (64 B from platform TRNG when available), the 16-byte `myNodeInfo.device_id`, and a call to `random()` into the rweather/Crypto library's software RNG, then `Curve25519::dh1`. `regeneratePublicKey` recomputes the public half from a known private (used when restoring from backup). -- **Keygen entry points**: at boot, `NodeDB` calls `generateKeyPair` (or `regeneratePublicKey` when a stored private key is present and passes a low-entropy check) **directly** when `!owner.is_licensed` and `config.lora.region != UNSET`. `ensurePkiKeys` wraps the same logic for runtime/admin flows — it's the path `AdminModule::handleSetConfig` runs when first assigning a valid region or when security config is written; **do not assume it's the universal boot-time gate**, because the NodeDB path bypasses it. +- **Keygen entry points**: at boot, `NodeDB` calls `generateKeyPair` (or `regeneratePublicKey` when a stored private key is present and passes a low-entropy check) **directly** when `!owner.is_licensed` and `config.lora.region != UNSET`. `ensurePkiKeys` wraps the same logic for runtime/admin flows - it's the path `AdminModule::handleSetConfig` runs when first assigning a valid region or when security config is written; **do not assume it's the universal boot-time gate**, because the NodeDB path bypasses it. - **Handshake**: `Curve25519::dh2(local_private, remote_public) → 32-byte shared secret → SHA-256 → 32-byte AES-256 key`. Recomputed per packet. The SHA-256 step is effectively a KDF over the raw ECDH output. -- **Cipher**: AES-256-CCM via `aes_ccm_ae` / `aes_ccm_ad` (`src/mesh/aes-ccm.cpp`). MAC length (the `M` parameter) is **8 bytes**. No AAD — the MAC covers ciphertext only. +- **Cipher**: AES-256-CCM via `aes_ccm_ae` / `aes_ccm_ad` (`src/mesh/aes-ccm.cpp`). MAC length (the `M` parameter) is **8 bytes**. No AAD - the MAC covers ciphertext only. - **Nonce (13 bytes / 104 bit)**: `aes_ccm_ae`/`aes_ccm_ad` use a 13-byte CCM nonce (`L = 2` is hardcoded in `src/mesh/aes-ccm.cpp`), not a 16-byte nonce. For PKI packets, `CryptoEngine::initNonce(fromNode, packetNum, extraNonce)` starts from the usual packet-derived nonce material, then overwrites nonce bytes `4..7` with a fresh 32-bit `extraNonce = random()`. The effective nonce bytes are therefore: bytes `0..3` = `packet_id`, bytes `4..7` = transmitted `extraNonce`, bytes `8..11` = `from_node`, byte `12` = `0x00`. The receiver reconstructs the same 13-byte nonce from the packet metadata plus the appended `extraNonce`. - **Wire overhead**: 12 bytes appended to the ciphertext = 8-byte MAC ‖ 4-byte extraNonce. Defined as `MESHTASTIC_PKC_OVERHEAD = 12` in `src/mesh/RadioInterface.h`. Only the 4-byte `extraNonce` is sent; the rest of the 13-byte CCM nonce is reconstructed from packet fields as described above. The Router's send path checks this overhead against `MAX_LORA_PAYLOAD_LEN` before committing to PKI. -- **Send selection** (`Router::send`): the sender enters the PKI path when **all** hold — we're the originator AND not Ham mode AND not Portduino simradio AND not on the `serial`/`gpio` channels (unless the packet is already marked `pki_encrypted`) AND `config.security.private_key.size == 32` AND destination is a single node (not broadcast) AND the portnum isn't infrastructure. `TRACEROUTE_APP`, `NODEINFO_APP`, `ROUTING_APP`, and `POSITION_APP` are routed through channel encryption even when DMed (these need to be readable by relaying peers). Once on the PKI path, if the destination's public key isn't in our NodeDB the send **fails** with `PKI_SEND_FAIL_PUBLIC_KEY` — it does not silently fall back to channel encryption. If the client explicitly set `pki_encrypted=true` and any condition blocks PKI, the send fails with `PKI_FAILED`. +- **Send selection** (`Router::send`): the sender enters the PKI path when **all** hold - we're the originator AND not Ham mode AND not Portduino simradio AND not on the `serial`/`gpio` channels (unless the packet is already marked `pki_encrypted`) AND `config.security.private_key.size == 32` AND destination is a single node (not broadcast) AND the portnum isn't infrastructure. `TRACEROUTE_APP`, `NODEINFO_APP`, `ROUTING_APP`, and `POSITION_APP` are routed through channel encryption even when DMed (these need to be readable by relaying peers). Once on the PKI path, if the destination's public key isn't in our NodeDB the send **fails** with `PKI_SEND_FAIL_PUBLIC_KEY` - it does not silently fall back to channel encryption. If the client explicitly set `pki_encrypted=true` and any condition blocks PKI, the send fails with `PKI_FAILED`. - **Receive selection** (`Router::perhapsDecode`): try PKI decrypt first when `channel == 0` AND `isToUs(p)` AND not broadcast AND both peers have public keys in NodeDB AND `rawSize > MESHTASTIC_PKC_OVERHEAD`. On success the packet gets `pki_encrypted=true` stamped and the sender's public key copied into `p->public_key` for downstream authorization. ### Remote admin authorization Implemented in `src/modules/AdminModule.cpp` → `handleReceivedProtobuf`. The authorization check runs in this order: -1. **Response messages** — if `messageIsResponse(r)` is true (the payload is a response to one of our earlier admin requests), it's accepted without any further check. The in-file comment flags this as a known-untightened gap: a stricter implementation would remember which `public_key` we last queried and reject responses that don't match. -2. **Local admin** — `mp.from == 0` (phone app over BLE, serial CLI, internal module); never travels over the air. **Rejected** if `config.security.is_managed` is true, because managed devices expect admin to arrive over the air through an authorized remote path. -3. **Legacy admin channel (deprecated)** — the packet arrived on a channel named literally `"admin"`. Gated by `config.security.admin_channel_enabled`; returns `NOT_AUTHORIZED` if the flag is false. Kept for backward compatibility; new deployments should use PKI admin. -4. **PKI admin (preferred for remote)** — `mp.pki_encrypted == true` AND `mp.public_key` matches one of `config.security.admin_key[0..2]` (up to three authorized 32-byte Curve25519 public keys, typically copied from the admin node's own `user.public_key`). +1. **Response messages** - if `messageIsResponse(r)` is true (the payload is a response to one of our earlier admin requests), it's accepted without any further check. The in-file comment flags this as a known-untightened gap: a stricter implementation would remember which `public_key` we last queried and reject responses that don't match. +2. **Local admin** - `mp.from == 0` (phone app over BLE, serial CLI, internal module); never travels over the air. **Rejected** if `config.security.is_managed` is true, because managed devices expect admin to arrive over the air through an authorized remote path. +3. **Legacy admin channel (deprecated)** - the packet arrived on a channel named literally `"admin"`. Gated by `config.security.admin_channel_enabled`; returns `NOT_AUTHORIZED` if the flag is false. Kept for backward compatibility; new deployments should use PKI admin. +4. **PKI admin (preferred for remote)** - `mp.pki_encrypted == true` AND `mp.public_key` matches one of `config.security.admin_key[0..2]` (up to three authorized 32-byte Curve25519 public keys, typically copied from the admin node's own `user.public_key`). 5. **Fallthrough** → `NOT_AUTHORIZED`. -On top of authorization, any remote admin message that **mutates** state (not a request, not a response) also has to pass a session-key check (`checkPassKey`): the client must first pull a fresh 8-byte `session_passkey` via `get_admin_session_key_request`, then echo that passkey back in the mutating message. The device rotates the passkey after 150 s and rejects values older than 300 s — a narrow anti-replay window on top of the PKI layer. +On top of authorization, any remote admin message that **mutates** state (not a request, not a response) also has to pass a session-key check (`checkPassKey`): the client must first pull a fresh 8-byte `session_passkey` via `get_admin_session_key_request`, then echo that passkey back in the mutating message. The device rotates the passkey after 150 s and rejects values older than 300 s - a narrow anti-replay window on top of the PKI layer. -`config.security.is_managed = true` disables **local** admin writes (`mp.from == 0` is rejected). It does not by itself force every admin action through PKI — the legacy `"admin"` channel still authorizes remote admin when `config.security.admin_channel_enabled == true`. The AdminModule refuses to persist `is_managed=true` unless at least one `admin_key` is populated — a deliberate guard against operators locking themselves out. +`config.security.is_managed = true` disables **local** admin writes (`mp.from == 0` is rejected). It does not by itself force every admin action through PKI - the legacy `"admin"` channel still authorizes remote admin when `config.security.admin_channel_enabled == true`. The AdminModule refuses to persist `is_managed=true` unless at least one `admin_key` is populated - a deliberate guard against operators locking themselves out. ### Key-rotation hazards (actions that invalidate peers) - **`factory_reset_device`** (the "full" variant, calls `NodeDB::factoryReset(eraseBleBonds=true)`) → **wipes** the X25519 private key; a fresh keypair is generated on the next region-set. Every existing peer holds the old public key, so DMs to this node silently fail PKI decrypt until every peer re-exchanges NodeInfo. - **`factory_reset_config`** (the "partial" variant, calls `NodeDB::factoryReset()` with `eraseBleBonds=false`) → **preserves** the X25519 private key in `installDefaultConfig(preserveKey=true)`; the public key is zeroed and gets rebuilt from the preserved private key on the next boot via the NodeDB path's `regeneratePublicKey` call. Identity is preserved and the mesh does not need to re-exchange keys. - **`region=UNSET → valid region`** → `ensurePkiKeys` runs inside the same `handleSetConfig` path; missing keys get generated at that moment. -- **Ham mode transitions** — entering Ham mode (`user.is_licensed=true`) runs `Channels::ensureLicensedOperation`, which **wipes every channel PSK** (all traffic becomes cleartext) and disables the legacy admin channel. The X25519 private key is preserved on the device but not used because `Router::send` skips PKI when `owner.is_licensed` is true. Leaving Ham mode re-enables PKI with the preserved keypair but does not restore the wiped channel PSKs — the operator has to re-set them. +- **Ham mode transitions** - entering Ham mode (`user.is_licensed=true`) runs `Channels::ensureLicensedOperation`, which **wipes every channel PSK** (all traffic becomes cleartext) and disables the legacy admin channel. The X25519 private key is preserved on the device but not used because `Router::send` skips PKI when `owner.is_licensed` is true. Leaving Ham mode re-enables PKI with the preserved keypair but does not restore the wiped channel PSKs - the operator has to re-set them. - **Channel 0 PSK change** → every peer must re-learn the channel hash; cached NodeInfo becomes temporarily unreachable until the next broadcast. - **`security.private_key` blanked via admin** → regenerates both halves (unless in Ham mode) and propagates the new public key via NodeInfo. -## Project Structure +## NodeDB Layout (v25) + +`DEVICESTATE_CUR_VER = 25`, `DEVICESTATE_MIN_VER = 24`. The on-device NodeDB was split in v25 into a slim header table plus four optional satellite stores. Older v24 saves auto-migrate at boot. Old training-data instincts (`node->user.long_name`, `node->position.latitude_i`, `node->is_favorite`, `node->device_metrics.battery_level`) are wrong now - the fields aren't there. Read this section before touching anything that walks `nodeDB->meshNodes`. + +### Slim `NodeInfoLite` + +`UserLite` is flattened onto `NodeInfoLite` (no nested sub-message); `position` and `device_metrics` are removed entirely (tags reserved). MAC address is dropped. Long names are capped at 25 chars (`max_size:25` in `deviceonly.options`); `hw_model` and `role` are `int_size:8`. Encoded size dropped from ~166 B → ~105 B per node. +Booleans are bit-packed into `NodeInfoLite.bitfield`. **Do not read or write the bits directly** - use the inline helpers in `src/mesh/NodeDB.h`: + +```cpp +nodeInfoLiteHasUser(n) // bit 5 - user fields populated +nodeInfoLiteIsFavorite(n) // bit 3 +nodeInfoLiteIsIgnored(n) // bit 4 +nodeInfoLiteIsMuted(n) // bit 1 +nodeInfoLiteIsLicensed(n) // bit 6 - Ham mode peer +nodeInfoLiteIsKeyManuallyVerified(n) // bit 0 +nodeInfoLiteHasIsUnmessagable(n) // bit 8 - "is_unmessagable was sent" +nodeInfoLiteIsUnmessagable(n) // bit 7 +// via_mqtt is bit 2 (mask exposed; predicate uses the mask directly) + +nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); // setter ``` + +### Satellite stores + +Four `std::unordered_map` members on `NodeDB`, each gated by its own build flag: + +| Map | Value type | Build flag | +| ----------------- | ------------------------------- | ---------------------------------- | +| `nodePositions` | `meshtastic_PositionLite` | `MESHTASTIC_EXCLUDE_POSITIONDB` | +| `nodeTelemetry` | `meshtastic_DeviceMetrics` | `MESHTASTIC_EXCLUDE_TELEMETRYDB` | +| `nodeEnvironment` | `meshtastic_EnvironmentMetrics` | `MESHTASTIC_EXCLUDE_ENVIRONMENTDB` | +| `nodeStatus` | `meshtastic_StatusMessage` | `MESHTASTIC_EXCLUDE_STATUSDB` | + +Defaults are ON (i.e., maps **excluded**) for STM32WL only - see `src/mesh/mesh-pb-constants.h`. On every other arch all four maps are present. When excluded, the map member is absent and the corresponding accessors return `false`. + +All four maps are guarded by **`mutable concurrency::Lock satelliteMutex`** - concurrent access from receive threads, the phone API state machine, and the renderer is the rule, not the exception. + +### Accessor convention + +**Never hand out pointers into the maps.** Use the copy-out accessors on `NodeDB`: + +```cpp +bool copyNodePosition(NodeNum, meshtastic_PositionLite &out) const; +bool copyNodeTelemetry(NodeNum, meshtastic_DeviceMetrics &out) const; +bool copyNodeEnvironment(NodeNum, meshtastic_EnvironmentMetrics &out) const; +bool copyNodeStatus(NodeNum, meshtastic_StatusMessage &out) const; +``` + +Each takes the lock, copies the value if present, returns `false` if the entry is absent or the DB is excluded. Pass-by-out-param is deliberate - pointer-style accessors would invite UAF and lock-leak bugs across the renderer. The "has any X" convenience predicates (`hasValidPosition` etc.) are implemented in terms of these. + +Writers go through `setNodeStatus`, `updatePosition`, `updateTelemetry` (which dispatches on `which_variant` for device vs environment metrics) - these own the lock and the eviction hooks. + +### Eviction + +Every code path that drops a node from the header table must also evict the satellites. The single chokepoint is `eraseNodeSatellites(NodeNum)`; it's already called from `getOrCreateMeshNode`'s oldest-boring eviction, `demoteOldestHotNodesToWarm` (the over-cap warm-tier migration), `removeNodeByNum`, both branches of `resetNodes`, `cleanupMeshDB`, `addFromContact`'s ignored-branch, and `AdminModule`'s `set_ignored_node`. Add new eviction sites here, not by calling `.erase()` directly. (Note: `enforceSatelliteCaps`/`evictSatelliteOverCap` call `.erase()` directly on purpose - that's a satellite-only cap trim where the node _stays_ in the header, a different operation from this chokepoint.) + +### Warm tier (long-tail identity) + +On every arch except STM32WL and bare nRF52832 (`WARM_NODE_COUNT > 0`), a node evicted from the header table is not forgotten outright: `WarmNodeStore` (`src/mesh/WarmNodeStore.{h,cpp}`) keeps a 40 B `{num, last_heard, public_key}` record per evicted node - primarily so PKI DMs to/from a long-tail node keep decrypting without re-running a NodeInfo exchange (the rest of `NodeInfoLite` rebuilds from traffic in seconds). + +- **Write:** `getOrCreateMeshNode`'s eviction and `demoteOldestHotNodesToWarm` (the over-cap boot migration) call `warmStore.absorb(num, last_heard, key)` _before_ the node leaves the header. +- **Read-back:** `getOrCreateMeshNode` calls `warmStore.take()` to rehydrate `last_heard` + key when a warm node is re-admitted; `copyPublicKey()` falls back to the warm tier so the PKI send path finds keys for evicted peers. +- **Persistence:** nRF52840 uses a 12 KB raw-flash record-ring at `0xEA000` (below LittleFS; append + replay + compact-on-rotate, link-guarded by `nrf52840_s140_v7.ld` and `extra_scripts/nrf52_warm_region.py`). Everywhere else: a `/prefs/warm.dat` snapshot flushed by `saveIfDirty()` on the node-DB save cadence. +- **Tunables** (`mesh-pb-constants.h`): `WARM_NODE_COUNT` (per-arch; `0` disables the tier) and `MAX_NUM_NODES` (hot cap - 120 on nRF52840/generic ESP32 to fit the 28 KB LittleFS; ESP32-S3 keeps its flash-scaled 100/200/250, portduino 250). Verbose migration/self-care tracing routes through `LOG_MIGRATION`, gated by `MESHTASTIC_NODEDB_MIGRATION_VERBOSE`. + +### Satellite caps + +Only the freshest `MAX_SATELLITE_NODES` nodes keep satellite payloads; the rest of the header table carries just the `NodeInfoLite`. The cap is **per-platform**: 40 on RAM-constrained parts (nRF52840, generic ESP32) since the four maps live in internal SRAM (not PSRAM, ~408 B/node across the four), and 250 on flash-rich hosts (ESP32-S3, portduino) so every hot node can carry rich data as before the cap existed. `enforceSatelliteCaps()` trims each map to the cap on load (returns whether it trimmed); `evictSatelliteOverCap()` trims before each insert. Eviction is by the owning node's hot `last_heard` (stalest first, demoted/absent nodes rank as `last_heard==0`); self is never trimmed. + +### On-boot self-care + +`NodeDB::nodeDBSelfCare()` runs once identity is established (the constructor after key (re)gen, and `reloadFromDisk()` - _not_ inside `loadFromDisk`, where `getNodeNum()` is still 0). It confirms self is present (warns if a non-empty DB is missing us - a foreign/over-cap file), pins self to index 0, demotes/trims only **non-self** overflow into the warm tier, then rewrites `nodes.proto` **once** and only if it healed something - and never while encrypted storage is locked (it would persist placeholder defaults). `loadFromDisk` deliberately leaves the loaded store untrimmed for this pass. + +### Sync flow: thin NodeInfo + post-COMPLETE_ID replay (no opt-in) + +There is no capability flag and no special "gradient" nonce. The **default** sync flow is: + +1. Config / module-config / channel / metadata segments (same as before). +2. `STATE_SEND_OWN_NODEINFO` - **our own** NodeInfo, still bundled with our position and device_metrics (because the replay snapshot excludes our own NodeNum). Emitted via `ConvertToNodeInfo(lite)`. +3. `STATE_SEND_OTHER_NODEINFOS` - every other peer's NodeInfo, **always thin** (no `position`, no `device_metrics`). Emitted via `ConvertToNodeInfoThin(lite)`. +4. `STATE_SEND_FILEMANIFEST` → `STATE_SEND_COMPLETE_ID` - the phone sees `config_complete_id` and treats sync as done. +5. `STATE_SEND_PACKETS` - live mesh packets, with a trailing replay drain interleaved. The replay drain walks four cached satellite stores in order (positions → telemetry → environment → status) and emits each cached entry as an ordinary `MeshPacket` on the matching portnum (`POSITION_APP`, `TELEMETRY_APP` device + environment variants, `NODE_STATUS_APP`). These are indistinguishable on the wire from live mesh traffic, so clients need no special handling - any code that already updates UI on `POSITION_APP` etc. works. + +`PhoneAPI::sendConfigComplete()` arms `replayPhase = REPLAY_PHASE_POSITIONS` for default/full sync and `SPECIAL_NONCE_ONLY_NODES`, while `SPECIAL_NONCE_ONLY_CONFIG` skips replay. The drain runs inside `STATE_SEND_PACKETS` via `popReplayPacket()`, lower priority than live traffic. When all four phases drain, `replayPhase` flips back to `REPLAY_PHASE_IDLE` and the snapshot vectors get `shrink_to_fit`ed. + +STM32WL and any other build with all four `MESHTASTIC_EXCLUDE_*DB` flags set produces zero replay packets - `popReplayPacket` advances through each phase in microseconds without emitting anything. + +Special nonces that still mean something: + +- `SPECIAL_NONCE_ONLY_CONFIG` (69420) - skip node sync entirely, just config. +- `SPECIAL_NONCE_ONLY_NODES` (69421) - skip config segments, jump straight to `STATE_SEND_OWN_NODEINFO`. Still gets the post-COMPLETE_ID replay drain. + +There are no other reserved nonces; everything else is a fresh random `want_config_id` from the client. + +### v24 → v25 migration + +The legacy migration code lives in **`src/mesh/NodeDBLegacyMigration.cpp`**, not in `NodeDB.cpp`. It owns the `meshtastic_NodeDatabase_Legacy` callback and `NodeDB::migrateLegacyNodeDatabase()`. The legacy proto descriptor is `protobufs/meshtastic/deviceonly_legacy.proto` (only included by the migration TU). The boot path peeks the file's leading version tag, runs the migration if `version < 25`, then re-saves in v25 layout. The legacy descriptor is scheduled for removal once `DEVICESTATE_MIN_VER` is bumped. + +### Read-site rules of thumb + +- Never `node->position.X` / `node->device_metrics.X` - those fields no longer exist. Pull from the satellite map via `copyNodePosition` / `copyNodeTelemetry`. +- Never `node->user.long_name` - `long_name`, `short_name`, `public_key`, `hw_model`, `role`, `macaddr` (gone), `is_licensed`, `is_unmessagable` are flat on `NodeInfoLite`. +- Never `node->is_favorite` / `node->is_ignored` / `node->via_mqtt` / `node->is_key_manually_verified` - use the bitfield helpers. +- Never assume `nodeDB->getMeshNode(num)->position.time` - call `copyNodePosition` and check the return. +- Don't lock `satelliteMutex` yourself in renderer code; the copy-out accessors already do. + +Unit tests for the conversion layer live in `test/test_type_conversions/test_main.cpp` (Unity) - bitfield round-trips, `long_name` truncation, thin-vs-full conversions. Add cases there when extending the schema. + +## Project Structure + +```text firmware/ ├── src/ # Main source code │ ├── main.cpp # Application entry point @@ -190,14 +318,25 @@ firmware/ ## Coding Conventions +### Formatting & the trunk toolchain + +`trunk fmt` is the project formatter (`trunk_check` CI rejects unformatted code). For Claude Code users, `.claude/settings.json` ships a PostToolUse hook that runs `trunk fmt --force` on every file the agent writes or edits. The hook is pure sh/grep/sed - no python or jq required - but trunk itself must be able to run: + +- Trunk's launcher (`~/.cache/trunk/launcher/trunk`, or `trunk` on PATH) downloads the CLI version pinned in `.trunk/trunk.yaml` on first use and again whenever that pin is bumped. **The launcher needs `curl` or `wget`**; without one it fails with "Cannot download… please install curl or wget", and the hook surfaces that as a warning on every write. +- No curl/wget available (e.g. a minimal WSL image)? Bootstrap by hand with any Python (PlatformIO bundles one at `~/.platformio/penv/bin/python`): download `https://trunk.io/releases//trunk--linux-x86_64.tar.gz` and place the `trunk` binary at `~/.cache/trunk/cli/-linux-x86_64/trunk` (chmod +x), where `` is the `cli.version` from `.trunk/trunk.yaml`. +- The hook fails loudly by design (visible warning, non-blocking). Silent no-op formatting hooks hide real breakage - don't re-add `2>/dev/null || true` around the whole thing. +- More generally: don't assume a stock Linux userland in hooks or helper scripts - minimal WSL/container images may lack `python3`, `curl`, `wget`, and `jq`. Prefer plain sh + coreutils, or PlatformIO's bundled Python for anything heavier. + ### General Style - Follow existing code style - run `trunk fmt` before commits - Prefer `LOG_DEBUG`, `LOG_INFO`, `LOG_WARN`, `LOG_ERROR` for logging +- **Format node IDs and packet IDs as `0x%08x` in logs.** This covers `NodeNum`/`PacketId` and the `uint32_t` packet fields `from`, `to`, `id`, `dest`, `source`, `request_id`, and `node_id`. They are 32-bit, so 8 hex digits is exact - `%08x` never truncates or leaves a value ragged. Do **not** use `%x` (variable width) or `%0x` (a no-op typo for `%08x` - the `0` flag does nothing without a width). User-facing display uses `!%08x` (the `!xxxxxxxx` convention), e.g. `Applet::hexifyNodeNum`. +- **Do not zero-pad one-byte values to 8.** `next_hop`, `relay_node`, and the next-hop hint are `uint8_t` last-byte route hints, and `channel` is a one-byte hash/index - log these as `0x%x` (or `%d`). Padding a byte to `0x000000ab` falsely implies a full node number. The same goes for I2C addresses, register values, flags/bitmasks, and error/reason codes: they are not IDs, so leave them `0x%x`. - Use `assert()` for invariants that should never fail - C++17 features are available (`std::optional`, structured bindings, `if constexpr`, etc.) -- **Keep code comments minimal — one or two lines, max.** Comment only when the _why_ isn't obvious from the code; never restate what the next line does. No multi-paragraph block comments explaining straightforward changes. The diff and commit message carry the rationale; the code carries the behavior. -- **Use `Throttle` for time-based rate limiting, not raw `millis()` math.** `src/mesh/Throttle.h` provides `Throttle::isWithinTimespanMs(lastMs, intervalMs)` (returns true while inside the cooldown) and `Throttle::execute(&lastMs, intervalMs, func)` (function-pointer form that updates the timestamp on fire). Use these for any "did N ms pass since X" check — raw `millis() > lastMs + N` is rollover-unsafe (breaks after ~49.7 days) and inconsistent with the rest of the codebase. The helpers compute `now - lastMs` with unsigned subtraction, which wraps correctly. +- **Keep code comments minimal - one or two lines, max.** Comment only when the _why_ isn't obvious from the code; never restate what the next line does. No multi-paragraph block comments explaining straightforward changes. The diff and commit message carry the rationale; the code carries the behavior. +- **Use `Throttle` for time-based rate limiting, not raw `millis()` math.** `src/mesh/Throttle.h` provides `Throttle::isWithinTimespanMs(lastMs, intervalMs)` (returns true while inside the cooldown) and `Throttle::execute(&lastMs, intervalMs, func)` (function-pointer form that updates the timestamp on fire). Use these for any "did N ms pass since X" check - raw `millis() > lastMs + N` is rollover-unsafe (breaks after ~49.7 days) and inconsistent with the rest of the codebase. The helpers compute `now - lastMs` with unsigned subtraction, which wraps correctly. ### Naming Conventions @@ -287,7 +426,7 @@ Multiple display driver families in `src/graphics/`: **InkHUD** (`src/graphics/niche/InkHUD/`) is an event-driven e-ink UI framework: -- Applet-based architecture — modular display tiles +- Applet-based architecture - modular display tiles - Read-only, static display optimized for minimal refreshes and low power - Configured per-variant via `nicheGraphics.h` - Separate PlatformIO config: `src/graphics/niche/InkHUD/PlatformioConfig.ini` @@ -344,6 +483,7 @@ Key defines in variant.h: - Regenerate with `bin/regen-protos.sh` - Message types prefixed with `meshtastic_` - Nanopb `.options` files control field sizes and encoding +- **Never edit or commit files under `src/mesh/generated/`.** They are regenerated from the [`meshtastic/protobufs`](https://github.com/meshtastic/protobufs) submodule by the `update_protobufs.yml` GitHub Action and any hand edits will be overwritten - guaranteed merge conflict on the next sync. To change a wire format, open a PR against the protobufs repo first; the workflow then re-runs `bin/regen-protos.sh` and opens a PR here with the regenerated sources. ### Conditional Compilation @@ -363,7 +503,7 @@ Key defines in variant.h: ## Build System -## Agent Tooling Baseline +### Agent Tooling Baseline Mirror counterpart: `AGENTS.md` under **Agent Tooling Baseline**. @@ -416,7 +556,7 @@ pio run -e native-macos # Build headless macOS meshtasticd (Homebrew prere 1. Create directory under `variants///` 2. Add `variant.h` with pin definitions and hardware capability defines -3. Add `platformio.ini` with build config — use `extends` to reference common base (e.g., `esp32s3_base`) +3. Add `platformio.ini` with build config - use `extends` to reference common base (e.g., `esp32s3_base`) 4. Set `custom_meshtastic_support_level = 1` (PR builds) or `2` (merge builds) 5. For e-ink displays, add `nicheGraphics.h` for InkHUD configuration @@ -516,34 +656,96 @@ Most workflows can be triggered manually via `workflow_dispatch` for testing. ### Native unit tests (C++) -Unit tests in `test/` directory with 12 test suites: +Unit tests in `test/` directory. The canonical suite count is in `test/native-suite-count` and is cross-checked on every full run. Current suites: -- `test_crypto/` - Cryptography -- `test_mqtt/` - MQTT integration -- `test_radio/` - Radio interface -- `test_mesh_module/` - Module framework -- `test_meshpacket_serializer/` - Packet serialization -- `test_transmit_history/` - Retransmission tracking +- `test_admin_radio/` - LoRa region/config validation and AdminModule dispatch - `test_atak/` - ATAK integration +- `test_crypto/` - Cryptography - `test_default/` - Default configuration +- `test_hop_scaling/` - Hop scaling histogram and required-hop logic - `test_http_content_handler/` - HTTP handling +- `test_mac_from_string/` - MAC address parsing +- `test_mesh_module/` - Module framework +- `test_meshpacket_serializer/` - Packet serialization +- `test_mqtt/` - MQTT integration +- `test_nexthop_routing/` - Next-hop routing logic +- `test_nodedb_blocked/` - NodeDB blocked-node handling +- `test_packet_history/` - Packet history tracking +- `test_packet_signing/` - Packet signing +- `test_position_module/` - Position module behaviour +- `test_position_precision/` - Position precision helpers +- `test_radio/` - Radio interface +- `test_rtc/` - RTC / time handling - `test_serial/` - Serial communication +- `test_traffic_management/` - Traffic management (dedup, rate-limit, hop-trim, role exceptions) +- `test_transmit_history/` - Retransmission tracking +- `test_type_conversions/` - NodeDB v25 type conversion (bitfield round-trips, NodeInfoLite) +- `test_utf8/` - UTF-8 utilities +- `test_warm_store/` - Warm-tier node store -Run with: `pio test -e native` +**Preferred run command - `bin/run-tests.sh`** (uses the `coverage` env with ASan/LSan sanitizers; emits a machine-readable verdict on the final line; update `test/native-suite-count` when adding or removing suites): + +```bash +./bin/run-tests.sh # all suites +./bin/run-tests.sh -f test_traffic_management # single suite (yields FILTERED, not GREEN) +``` + +Exit codes and verdicts (exact counts will vary; examples below are illustrative): + +| Exit | Verdict | Meaning | +| ---- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 0 | `GREEN` | All canonical suites ran, all passed, no ignored test cases | +| 1 | `RED` | At least one failure, build error, or sanitizer fault | +| 2 | `AMBER` | All that ran passed, but something was lost: a suite silently went missing on a full run, individual test cases were skipped (`TEST_IGNORE`), or `test/native-suite-count` disagrees with the `test/` directory count | +| 3 | `FILTERED` | A `-f` run completed cleanly; suites outside the filter were intentionally not run | + +Examples - exact counts will vary by suite count and env: + +```text +# GREEN: all suites ran and passed +RESULT: GREEN N/N suites passed [canonical: N/N] + +# RED: real test failure +RESULT: RED 1 failed + +# RED: sanitizer exit-time abort (all tests passed but process aborted at exit) +RESULT: RED exit-time abort (tests passed; likely sanitizer - see hint above) + +# AMBER: native-suite-count disagrees with test/ directory count (too low) +RESULT: AMBER test/ has 24 suite directories but native-suite-count says 5 - update test/native-suite-count after registering new suites + +# AMBER: native-suite-count disagrees with test/ directory count (too high) +RESULT: AMBER test/ has 24 suite directories but native-suite-count says 99 - update test/native-suite-count after removing suites + +# FILTERED: single suite run completed cleanly +RESULT: FILTERED 1/24 suites ran (not run: test_admin_radio test_atak …) - filtered: test_serial [canonical: 1/24] +``` + +> **Copilot interface note:** When running tests via the Copilot chat interface, edits made through the chat may not be reflected in the on-disk files that the test binary reads. If tests pass in chat but fail locally (or vice versa), verify the files on disk match what you expect before trusting the result. Always confirm with a local terminal run. + +Raw `pio test` (no sanitizers, no verdict logic) - use only when you need to override the env: + +```bash +~/.platformio/penv/bin/python -m platformio test -e native -f test_your_suite > /tmp/test_out.txt 2>&1 +grep -E 'error:|PASS|FAIL|succeeded|failed' /tmp/test_out.txt +tail -15 /tmp/test_out.txt +``` + +Do **not** pipe `pio test` - line-buffering makes the terminal appear hung and hides build errors. Simulation testing: `bin/test-simulator.sh` Quick entry point for new test modules: `test/README.md` (native unit-test authoring guide, skeleton, pitfalls, and setup checklist). -### Hardware-in-the-loop tests (`mcp-server/tests/`) +### Hardware-in-the-loop tests ([meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp)) -Separate pytest suite that exercises real USB-connected Meshtastic devices. See the **MCP Server & Hardware Test Harness** section below for invocation, tier layout, and agent usage rules. +Separate pytest suite that exercises real USB-connected Meshtastic devices. It now lives in the standalone [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) repo, run against a firmware checkout via `MESHTASTIC_FIRMWARE_ROOT`. See the **MCP Server & Hardware Test Harness** section below for invocation, tier layout, and agent usage rules. ## MCP Server & Hardware Test Harness -The `mcp-server/` directory houses a firmware-aware [MCP](https://modelcontextprotocol.io/) server plus a pytest-based integration suite. AI agents that speak MCP get a well-defined tool surface for flashing, configuring, and inspecting physical Meshtastic devices — use it instead of hand-rolling `pio` or `meshtastic --port` calls where possible. `mcp-server/README.md` is the operator-facing setup doc; this section is the agent-facing usage contract. +The firmware-aware [MCP](https://modelcontextprotocol.io/) server plus its pytest-based integration suite now live in the standalone [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) repo. AI agents that speak MCP get a well-defined tool surface for flashing, configuring, and inspecting physical Meshtastic devices - use it instead of hand-rolling `pio` or `meshtastic --port` calls where possible. The meshtastic-mcp repo's README is the operator-facing setup doc; this section is the agent-facing usage contract. -The repo registers the server via `.mcp.json` at the repo root — Claude Code picks it up automatically once `mcp-server/.venv/` is built (`cd mcp-server && python3 -m venv .venv && .venv/bin/pip install -e '.[test]'`). +The repo registers the server via `.mcp.json` at the repo root - Claude Code / Copilot pick it up automatically and run it through `uvx --from git+https://github.com/meshtastic/meshtastic-mcp meshtastic-mcp`, so the MCP tools work with no local build. To run the pytest hardware harness instead, clone [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) and point `MESHTASTIC_FIRMWARE_ROOT` at this firmware checkout. ### When to use which surface @@ -551,70 +753,80 @@ The repo registers the server via `.mcp.json` at the repo root — Claude Code p | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | Find a connected device | `mcp__meshtastic__list_devices` | | Read a live node's config/state | `mcp__meshtastic__device_info`, `list_nodes`, `get_config` | -| Mutate a device (owner, region, channels, reboot) | `set_owner`, `set_config`, `set_channel_url`, `reboot`, `shutdown`, `factory_reset` — all require `confirm=True` | +| Mutate a device (owner, region, channels, reboot) | `set_owner`, `set_config`, `set_channel_url`, `reboot`, `shutdown`, `factory_reset` - all require `confirm=True` | | Flash firmware to a variant | `pio_flash` (any arch) or `erase_and_flash` (ESP32 factory install) | | Stream serial logs while debugging | `serial_open` → `serial_read` loop → `serial_close` | | Administer `userPrefs.jsonc` build-time constants | `userprefs_get`, `userprefs_set`, `userprefs_reset`, `userprefs_manifest` | -| Run the regression suite | `./mcp-server/run-tests.sh` (or `/test` slash command) | +| Run the regression suite | `./run-tests.sh` from a meshtastic-mcp checkout (or `/test` slash command) | | Diagnose a specific device | `/diagnose [role]` slash command (read-only) | | Triage a flaky test | `/repro [count]` slash command | **One MCP call per port at a time.** `SerialInterface` holds an exclusive OS-level lock on the serial port for its lifetime. If a `serial_*` session is open on `/dev/cu.usbmodem101`, calling `device_info` on the same port will fail fast pointing at the active session. Sequence calls: open → read/mutate → close, then next device. Never parallelize tool calls on the same port. -### MCP tool surface (43 tools) +### MCP tool surface (44 tools) -Grouped by purpose. Full argument shapes in `mcp-server/README.md`; a few high-value signatures are called out here. +Grouped by purpose. Full argument shapes in the meshtastic-mcp repo's README; a few high-value signatures are called out here. - **Discovery & metadata**: `list_devices`, `list_boards`, `get_board` - **Build & flash**: `build`, `clean`, `pio_flash`, `erase_and_flash` (ESP32 only), `update_flash` (ESP32 OTA), `touch_1200bps` - **Serial sessions** (long-running, 10k-line ring buffer): `serial_open`, `serial_read`, `serial_list`, `serial_close` - **Device reads**: `device_info`, `list_nodes` -- **Device writes**: `set_owner`, `get_config`, `set_config`, `get_channel_url`, `set_channel_url`, `send_text`, `send_input_event` (inject a button/key press via the firmware's InputBroker), `set_debug_log_api`; destructive/power-state writes require `confirm=True`: `reboot`, `shutdown`, `factory_reset` +- **Device writes**: `set_owner`, `get_config`, `set_config`, `get_channel_url`, `set_channel_url`, `send_text`, `send_input_event` (inject a button/key press via the firmware's InputBroker), `inject_frame` (inject an over-the-air-style frame into the RX pipeline - see below), `set_debug_log_api`; destructive/power-state writes require `confirm=True`: `reboot`, `shutdown`, `factory_reset` - **userPrefs admin** (build-time constants, not runtime config): `userprefs_get`, `userprefs_set`, `userprefs_reset`, `userprefs_manifest`, `userprefs_testing_profile` - **Vendor escape hatches**: `esptool_chip_info`, `esptool_erase_flash`, `esptool_raw`, `nrfutil_dfu`, `nrfutil_raw`, `picotool_info`, `picotool_load`, `picotool_raw` - **USB power control** (via `uhubctl`, per-port PPPS toggle): `uhubctl_list` (read-only), `uhubctl_power(action='on'|'off', confirm=True)`, `uhubctl_cycle(delay_s, confirm=True)`. Target by raw `(location, port)` or by `role` (`"nrf52"`, `"esp32s3"`); role lookup checks `MESHTASTIC_UHUBCTL_LOCATION_` + `_PORT_` env vars first, falls back to VID auto-detection. -- **Observability** (UI tier + operator ad-hoc): `capture_screen(role, ocr=True)` — grabs a USB-webcam frame of the device OLED and optionally OCRs it. Requires `mcp-server[ui]` extras (`opencv-python-headless`, `easyocr`) and `MESHTASTIC_UI_CAMERA_DEVICE_` env var; falls through to a 1×1 black PNG `NullBackend` when unconfigured. +- **Observability** (UI tier + operator ad-hoc): `capture_screen(role, ocr=True)` - grabs a USB-webcam frame of the device OLED and optionally OCRs it. Requires `meshtastic-mcp[ui]` extras (`opencv-python-headless`, `easyocr`) and `MESHTASTIC_UI_CAMERA_DEVICE_` env var; falls through to a 1×1 black PNG `NullBackend` when unconfigured. + +`confirm=True` is a tool-level gate on top of whatever permission prompt your MCP host shows. **Don't bypass it** by asking the host to auto-approve - it exists specifically because MCP hosts sometimes remember "always allow this tool" and that's dangerous for `factory_reset`, `erase_and_flash`, `uhubctl_power(action='off')`, and `uhubctl_cycle`. + +**TCP / native-host nodes.** Setting `MESHTASTIC_MCP_TCP_HOST=` makes `list_devices` surface a `meshtasticd` daemon (e.g. the `native-macos` build) as a synthetic `tcp://host:port` entry, and `connect()` routes through `meshtastic.tcp_interface.TCPInterface` instead of `SerialInterface`. Every read/write/admin tool that flows through `connect()` works against the daemon transparently. USB-only tools (`pio_flash`, `erase_and_flash`, `update_flash`, `touch_1200bps`, `serial_open`, `esptool_*`, `nrfutil_*`, `picotool_*`) raise a clear `ConnectionError` when handed a `tcp://` port; `pio_flash` against a `native*` env raises a `FlashError` (no upload step - use `build` and run the binary directly). The pytest harness still assumes USB-attached devices per role; TCP-aware fixtures are deferred. See the meshtastic-mcp repo's README § "TCP / native-host nodes". + +### Frame injection: testing the off-air receive path -`confirm=True` is a tool-level gate on top of whatever permission prompt your MCP host shows. **Don't bypass it** by asking the host to auto-approve — it exists specifically because MCP hosts sometimes remember "always allow this tool" and that's dangerous for `factory_reset`, `erase_and_flash`, `uhubctl_power(action='off')`, and `uhubctl_cycle`. +The `toRadio` API can only inject **locally-originated** traffic - the firmware forces `p.from = 0` in `MeshService::handleToRadio`, which bypasses the `from != 0` receive path and everything gated on it (remote admin authorization, the admin session-passkey check, hop handling, promiscuous sniffing). To exercise those paths you either need a second transmitting radio, or **frame injection**: a build-flagged seam that delivers a client-supplied frame into the real RX pipeline as if it arrived off the LoRa chip. -**TCP / native-host nodes.** Setting `MESHTASTIC_MCP_TCP_HOST=` makes `list_devices` surface a `meshtasticd` daemon (e.g. the `native-macos` build) as a synthetic `tcp://host:port` entry, and `connect()` routes through `meshtastic.tcp_interface.TCPInterface` instead of `SerialInterface`. Every read/write/admin tool that flows through `connect()` works against the daemon transparently. USB-only tools (`pio_flash`, `erase_and_flash`, `update_flash`, `touch_1200bps`, `serial_open`, `esptool_*`, `nrfutil_*`, `picotool_*`) raise a clear `ConnectionError` when handed a `tcp://` port; `pio_flash` against a `native*` env raises a `FlashError` (no upload step — use `build` and run the binary directly). The pytest harness still assumes USB-attached devices per role; TCP-aware fixtures are deferred. See `mcp-server/README.md` § "TCP / native-host nodes". +- **Firmware:** build with `-D MESHTASTIC_ENABLE_FRAME_INJECTION=1` (`src/configuration.h`, off by default - it forges over-the-air traffic and must never ship enabled). `MeshService::injectAsReceived` extends the existing portduino `SimRadio` `SIMULATOR_APP` path to real hardware: it unwraps a `Compressed` envelope (`portnum == UNKNOWN_APP` → verbatim ciphertext the router decrypts; else → decoded payload for that portnum), then calls `router->enqueueReceivedMessage()` - the exact entry point `RadioLibInterface::handleReceiveInterrupt` uses. Injection is reached before the `p.from = 0` line, so a forged sender survives; `from == 0` is dropped to match real RX. +- **Host:** drive it with the meshtastic-mcp `inject_frame` tool (or `cli/meshinject.py`). The crafter replicates channel crypto (default-PSK expansion, `xorHash` channel hash, AES-CTR with the `packetId|from|0` nonce), so an encrypted frame decrypts on-device as if received. Modes: `text`, `raw`, `admin` (+ `pki`/`public_key` for the PKC-admin path), `ciphertext`, `fuzz` (malformed-frame decode-path/crash testing). +- **Example - remote-admin session-key repro:** set the target's `admin_key[0]` to a key you hold, then inject an `admin` set_owner with `pki=true`, that key, and a stale `session_hex`. The node logs `PKC admin payload with authorized sender key` → `Expected session key: 00…` → `Admin message without session_key!` - the exact ndoo scenario, on real silicon. Capture logs via `set_debug_log_api` on the same connection. +- **nRF52 gotcha:** the USB CDC wedges under rapid `SerialInterface` open/close churn (unrelated to injection) - keep setup + inject + log-capture on one connection; recover a hung board via a 1200 bps-touch DFU reflash. -### Hardware test suite (`mcp-server/run-tests.sh`) +### Hardware test suite (`run-tests.sh`, from a meshtastic-mcp checkout) The wrapper auto-detects connected devices (VID → role map: `0x239A` → `nrf52`, `0x303A`/`0x10C4` → `esp32s3`), maps each role to a PlatformIO env (`nrf52` → `rak4631`, `esp32s3` → `heltec-v3`, overridable via `MESHTASTIC_MCP_ENV_`), then invokes pytest. Zero pre-flight config needed from the operator. Suite tiers (collected + run in this order via `pytest_collection_modifyitems`): -1. `tests/unit/` — pure Python (boards parse, pio wrapper, userPrefs parse, testing profile, uhubctl parser). No hardware. -2. `tests/test_00_bake.py` — flashes each detected device with current `userPrefs.jsonc` merged with the session's test profile. Has its own skip-if-already-baked check comparing region + primary channel to the session profile; skips cheaply on warm devices. -3. `tests/mesh/` — multi-device mesh: bidirectional send, broadcast delivery, direct-with-ACK, mesh formation within 60s. Parametrized `[nrf52->esp32s3]` and `[esp32s3->nrf52]`. Includes `test_peer_offline_recovery` which uses uhubctl to physically power off one peer mid-conversation (requires uhubctl; skips without). -4. `tests/telemetry/` — `DEVICE_METRICS_APP` broadcast timing. -5. `tests/monitor/` — boot-log panic check. -6. `tests/recovery/` — `uhubctl` power-cycle round-trip + NVS persistence across hard reset. Requires `uhubctl` installed and a PPPS-capable hub; entire tier auto-skips otherwise. -7. `tests/ui/` — input-broker-driven screen navigation with camera + OCR evidence. -8. `tests/fleet/` — PSK seed session isolation. -9. `tests/admin/` — channel URL roundtrip, owner persistence across reboot. -10. `tests/provisioning/` — region + modem + slot bake, admin key presence, `UNSET` region blocks TX, userPrefs survive factory reset. +1. `tests/unit/` - pure Python (boards parse, pio wrapper, userPrefs parse, testing profile, uhubctl parser). No hardware. +2. `tests/test_00_bake.py` - flashes each detected device with current `userPrefs.jsonc` merged with the session's test profile. Has its own skip-if-already-baked check comparing region + primary channel to the session profile; skips cheaply on warm devices. +3. `tests/mesh/` - multi-device mesh: bidirectional send, broadcast delivery, direct-with-ACK, mesh formation within 60s. Parametrized `[nrf52->esp32s3]` and `[esp32s3->nrf52]`. Includes `test_peer_offline_recovery` which uses uhubctl to physically power off one peer mid-conversation (requires uhubctl; skips without). +4. `tests/telemetry/` - `DEVICE_METRICS_APP` broadcast timing. +5. `tests/monitor/` - boot-log panic check. +6. `tests/recovery/` - `uhubctl` power-cycle round-trip + NVS persistence across hard reset. Requires `uhubctl` installed and a PPPS-capable hub; entire tier auto-skips otherwise. +7. `tests/ui/` - input-broker-driven screen navigation with camera + OCR evidence. +8. `tests/fleet/` - PSK seed session isolation. +9. `tests/admin/` - channel URL roundtrip, owner persistence across reboot. +10. `tests/provisioning/` - region + modem + slot bake, admin key presence, `UNSET` region blocks TX, userPrefs survive factory reset. Invocation patterns: ```bash -./mcp-server/run-tests.sh # full suite (auto-bake-if-needed) -./mcp-server/run-tests.sh --force-bake # reflash before testing -./mcp-server/run-tests.sh --assume-baked # skip bake (caller vouches for device state) -./mcp-server/run-tests.sh tests/mesh # one tier -./mcp-server/run-tests.sh tests/mesh/test_direct_with_ack.py # one file -./mcp-server/run-tests.sh -k telemetry # name filter +# run from a meshtastic-mcp checkout, with MESHTASTIC_FIRMWARE_ROOT=/path/to/firmware +./run-tests.sh # full suite (auto-bake-if-needed) +./run-tests.sh --force-bake # reflash before testing +./run-tests.sh --assume-baked # skip bake (caller vouches for device state) +./run-tests.sh tests/mesh # one tier +./run-tests.sh tests/mesh/test_direct_with_ack.py # one file +./run-tests.sh -k telemetry # name filter ``` -**No hardware detected?** The wrapper auto-narrows to `tests/unit/` only and prints `detected hub : (none)` in the pre-flight header. Agents interpreting the output should call this out explicitly — a 52-test green run without hardware is qualitatively different from a 12-unit-test green run. +**No hardware detected?** The wrapper auto-narrows to `tests/unit/` only and prints `detected hub : (none)` in the pre-flight header. Agents interpreting the output should call this out explicitly - a 52-test green run without hardware is qualitatively different from a 12-unit-test green run. **Artifacts every run produces:** -- `mcp-server/tests/report.html` — self-contained pytest-html. Each test gets a `Meshtastic debug` section with the tail of firmware log + device state dump. **Open this first** on failures; it's the canonical evidence source. -- `mcp-server/tests/junit.xml` — CI-parseable. -- `mcp-server/tests/reportlog.jsonl` — pytest-reportlog stream (`$report_type` keyed JSONL). Consumed by the live TUI. -- `mcp-server/tests/fwlog.jsonl` — firmware log mirror from the `meshtastic.log.line` pubsub topic. Populated by the `_firmware_log_stream` autouse session fixture. +- `tests/report.html` - self-contained pytest-html. Each test gets a `Meshtastic debug` section with the tail of firmware log + device state dump. **Open this first** on failures; it's the canonical evidence source. +- `tests/junit.xml` - CI-parseable. +- `tests/reportlog.jsonl` - pytest-reportlog stream (`$report_type` keyed JSONL). Consumed by the live TUI. +- `tests/fwlog.jsonl` - firmware log mirror from the `meshtastic.log.line` pubsub topic. Populated by the `_firmware_log_stream` autouse session fixture. ### Live TUI (`meshtastic-mcp-test-tui`) @@ -634,7 +846,7 @@ A Textual-based live view that wraps `run-tests.sh`. Tails reportlog for per-tes Launch: ```bash -cd mcp-server +# from a meshtastic-mcp checkout (MESHTASTIC_FIRMWARE_ROOT set) .venv/bin/meshtastic-mcp-test-tui # full suite .venv/bin/meshtastic-mcp-test-tui tests/mesh # args pass through to pytest ``` @@ -659,29 +871,29 @@ House rules for agents running these prompts: ### Key fixtures (test authors + agents debugging) -`mcp-server/tests/conftest.py` provides: +`tests/conftest.py` (in the meshtastic-mcp checkout) provides: -- **`_session_userprefs`** (autouse session) — snapshots `userPrefs.jsonc` at session start, merges the session test profile via `userprefs.merge_active(test_profile)`, restores at teardown. Four layers of safety: pytest teardown + `atexit` + sidecar file (`userPrefs.jsonc.mcp-session-bak`) + startup self-heal in `run-tests.sh`. **Do not edit `userPrefs.jsonc` from inside a test.** -- **`_firmware_log_stream`** (autouse session) — subscribes to `meshtastic.log.line` pubsub on every connected `SerialInterface` and mirrors lines to `tests/fwlog.jsonl`. Drives the TUI firmware-log pane. -- **`_debug_log_buffer`** (autouse per-test) — captures last 200 firmware log lines + device state for attachment to the pytest-html `Meshtastic debug` section on failure. -- **`hub_devices`** (session) — `dict[role, SerialInterface]` with session-long exclusive port locks. Reason the TUI's device poller is gated to startup + post-run only. -- **`baked_mesh`** — parametrized mesh-pair fixture; depends on `test_00_bake`. `pytest_generate_tests` in `conftest.py` auto-generates `[nrf52->esp32s3]` and `[esp32s3->nrf52]` variants. -- **`test_profile`** — session-scoped dict: region, primary channel, admin key, PSK seed. Derived from `MESHTASTIC_MCP_SEED` (defaults to `mcp--`). +- **`_session_userprefs`** (autouse session) - snapshots `userPrefs.jsonc` at session start, merges the session test profile via `userprefs.merge_active(test_profile)`, restores at teardown. Four layers of safety: pytest teardown + `atexit` + sidecar file (`userPrefs.jsonc.mcp-session-bak`) + startup self-heal in `run-tests.sh`. **Do not edit `userPrefs.jsonc` from inside a test.** +- **`_firmware_log_stream`** (autouse session) - subscribes to `meshtastic.log.line` pubsub on every connected `SerialInterface` and mirrors lines to `tests/fwlog.jsonl`. Drives the TUI firmware-log pane. +- **`_debug_log_buffer`** (autouse per-test) - captures last 200 firmware log lines + device state for attachment to the pytest-html `Meshtastic debug` section on failure. +- **`hub_devices`** (session) - `dict[role, SerialInterface]` with session-long exclusive port locks. Reason the TUI's device poller is gated to startup + post-run only. +- **`baked_mesh`** - parametrized mesh-pair fixture; depends on `test_00_bake`. `pytest_generate_tests` in `conftest.py` auto-generates `[nrf52->esp32s3]` and `[esp32s3->nrf52]` variants. +- **`test_profile`** - session-scoped dict: region, primary channel, admin key, PSK seed. Derived from `MESHTASTIC_MCP_SEED` (defaults to `mcp--`). ### Firmware integration points tied to the test harness Two firmware changes exist specifically so the test harness works reliably. **Keep these in mind when touching related code.** -- **`src/mesh/StreamAPI.cpp` + `StreamAPI.h`** — `emitLogRecord` uses a dedicated `fromRadioScratchLog` + `txBufLog` pair and a `concurrency::Lock streamLock`. Before this fix, `debug_log_api_enabled=true` would tear `FromRadio` protobufs on the serial transport because `emitTxBuffer` and `emitLogRecord` shared a single scratch buffer. The conftest enables the log stream session-wide; without this fix the device would corrupt its own FromRadio replies mid-session. -- **`src/mesh/PhoneAPI.cpp`** — `ToRadio` `Heartbeat(nonce=1)` triggers `nodeInfoModule->sendOurNodeInfo(NODENUM_BROADCAST, true, 0, true)` for serial clients, mirroring the pre-existing behavior for TCP/UDP clients in `PacketAPI.cpp`. The mesh tests rely on this to force a NodeInfo broadcast right after connect so the peer discovers them before the test's first assertion. +- **`src/mesh/StreamAPI.cpp` + `StreamAPI.h`** - `emitLogRecord` uses a dedicated `fromRadioScratchLog` + `txBufLog` pair and a `concurrency::Lock streamLock`. Before this fix, `debug_log_api_enabled=true` would tear `FromRadio` protobufs on the serial transport because `emitTxBuffer` and `emitLogRecord` shared a single scratch buffer. The conftest enables the log stream session-wide; without this fix the device would corrupt its own FromRadio replies mid-session. +- **`src/mesh/PhoneAPI.cpp`** - `ToRadio` `Heartbeat(nonce=1)` triggers `nodeInfoModule->sendOurNodeInfo(NODENUM_BROADCAST, true, 0, true)` for serial clients, mirroring the pre-existing behavior for TCP/UDP clients in `PacketAPI.cpp`. The mesh tests rely on this to force a NodeInfo broadcast right after connect so the peer discovers them before the test's first assertion. -If you're modifying `StreamAPI`, `PhoneAPI`, `NodeInfoModule`, or `userPrefs` flow, run `./mcp-server/run-tests.sh` at minimum before asking for review. +If you're modifying `StreamAPI`, `PhoneAPI`, `NodeInfoModule`, or `userPrefs` flow, run `./run-tests.sh` (from a meshtastic-mcp checkout, with `MESHTASTIC_FIRMWARE_ROOT` pointed here) at minimum before asking for review. ### Recovery playbooks | Symptom | First check | Fix | | --------------------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `userPrefs.jsonc` dirty after test run | `git status --porcelain userPrefs.jsonc` | If non-empty, re-run `./mcp-server/run-tests.sh` once — the pre-flight self-heal restores from sidecar. If still dirty, `git checkout userPrefs.jsonc`. | +| `userPrefs.jsonc` dirty after test run | `git status --porcelain userPrefs.jsonc` | If non-empty, re-run `./run-tests.sh` (from a meshtastic-mcp checkout) once - the pre-flight self-heal restores from sidecar. If still dirty, `git checkout userPrefs.jsonc`. | | Port busy / wedged CP2102 on macOS | `lsof /dev/cu.usbserial-0001` | Kill the holder. USB replug if the kernel still reports busy. Often a stale `pio device monitor` or zombie `meshtastic_mcp` process. | | nRF52 appears unresponsive | `list_devices` shows VID `0x239A` but `device_info` times out | `touch_1200bps(port=...)` drops it into the DFU bootloader → `pio_flash` re-installs. | | Device fully wedged (Guru Meditation, frozen CDC, no DFU) | `list_devices` shows the VID but every admin call times out | `uhubctl_cycle(role="nrf52", confirm=True)` hard-power-cycles the port via USB hub PPPS. `baked_single`'s auto-recovery hook does this once automatically if uhubctl is installed. Falls back to physical replug if no PPPS hub. | @@ -691,17 +903,17 @@ If you're modifying `StreamAPI`, `PhoneAPI`, `NodeInfoModule`, or `userPrefs` fl | Entire `tests/recovery/` tier skipped | `command -v uhubctl` | Expected if `uhubctl` isn't on PATH. Install via `brew install uhubctl` (macOS) or `apt install uhubctl` (Debian/Ubuntu). Also skips if no hub advertises PPPS. | | Entire `tests/ui/` tier skipped ("firmware not baked with USERPREFS_UI_TEST_LOG") | reportlog.jsonl for the skip reason | Re-run with `--force-bake` so the UI-log macro gets compiled into the fresh firmware. First run after the Round-3 landing always re-bakes. | | `tests/ui/` runs but captures are all 1×1 black PNGs | `MESHTASTIC_UI_CAMERA_DEVICE_ESP32S3` | Env var not set → `NullBackend`. Point a USB webcam at the heltec-v3 OLED and set the device index; `.venv/bin/python -c "import cv2; [print(i, cv2.VideoCapture(i).read()[0]) for i in range(5)]"` discovers it. | -| Tests fail only on first attempt then pass on rerun | — | State leak from a prior session. Run with `--force-bake` to reset to a known state. | +| Tests fail only on first attempt then pass on rerun | - | State leak from a prior session. Run with `--force-bake` to reset to a known state. | ### Never do these without asking -- `factory_reset` — wipes node identity; regenerates PKI keypair. Mesh peers will reject old DMs until re-exchange. Legitimate only when the operator explicitly wants it. -- `erase_and_flash` — full chip erase; destroys all on-device state. -- `esptool_erase_flash` / `esptool_raw` write/erase — bypasses pio's safety chain. -- `set_config` on `lora.region` — changes regulatory domain; requires physical-location context the operator has and the agent doesn't. -- `reboot` / `shutdown` mid-test — breaks fixture invariants. +- `factory_reset` - wipes node identity; regenerates PKI keypair. Mesh peers will reject old DMs until re-exchange. Legitimate only when the operator explicitly wants it. +- `erase_and_flash` - full chip erase; destroys all on-device state. +- `esptool_erase_flash` / `esptool_raw` write/erase - bypasses pio's safety chain. +- `set_config` on `lora.region` - changes regulatory domain; requires physical-location context the operator has and the agent doesn't. +- `reboot` / `shutdown` mid-test - breaks fixture invariants. - `push -f`, `rebase -i`, `reset --hard`, or any history-rewriting git operation. -- Clicking computer-use tools on web links in Mail/Messages/PDFs — open URLs via the claude-in-chrome MCP so the extension's link-safety checks apply. +- Clicking computer-use tools on web links in Mail/Messages/PDFs - open URLs via the claude-in-chrome MCP so the extension's link-safety checks apply. ## Resources diff --git a/.github/prompts/mcp-diagnose.prompt.md b/.github/prompts/mcp-diagnose.prompt.md deleted file mode 100644 index 1049858f8ef..00000000000 --- a/.github/prompts/mcp-diagnose.prompt.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -mode: agent -description: Device health report via the meshtastic MCP tools (Copilot equivalent of the Claude Code /diagnose slash command) ---- - -# `/mcp-diagnose` — device health report - -Equivalent of `.claude/commands/diagnose.md`. Use when the operator asks to "check the devices", "what's the mesh looking like", "is nrf52 alive", etc. - -This prompt assumes the meshtastic MCP server is registered with your VS Code Copilot agent. If it isn't, fall back to running `./mcp-server/run-tests.sh tests/unit` plus a short `device_info` script via the terminal. - -## What to do - -1. **Enumerate hardware** via the `list_devices` MCP tool (with `include_unknown=True`). For each entry where `likely_meshtastic=True`, capture `port`, `vid`, `pid`, `description`. - -2. **Apply the operator's filter** (if any): - - No filter → every likely-meshtastic device. - - `nrf52` → `vid == 0x239a` - - `esp32s3` → `vid == 0x303a` or `vid == 0x10c4` - - A `/dev/cu.*` path → only that port. - - Anything else → substring match on port. - -3. **For each selected device, in sequence (don't parallelize — SerialInterface holds an exclusive port lock):** - - `device_info(port=

)` → `my_node_num`, `long_name`, `short_name`, `firmware_version`, `hw_model`, `region`, `num_nodes`, `primary_channel` - - `list_nodes(port=

)` → peer count, which peers have `publicKey`, SNR/RSSI distribution - - `get_config(section="lora", port=

)` → region, preset, channel_num, tx_power, hop_limit - - If anything looks off (can't connect, `num_nodes` wrong, missing `firmware_version`), open a short firmware-log window: `serial_open(port=

, env=)`, wait 3 seconds, `serial_read(session_id, max_lines=100)`, `serial_close(session_id)`. Infer env from VID (0x239a → `rak4631`, 0x303a/0x10c4 → `heltec-v3`) unless an `MESHTASTIC_MCP_ENV_` env var overrides it. - -4. **Hub health** (call once, not per-device): `uhubctl_list()` — enumerates every USB hub the host sees. Cross-reference each Meshtastic device's VID to find which hub + port it's on. Flag in the report if: - - No hub advertises `ppps=true` → `tests/recovery/` can't run; hard-recovery via `uhubctl_cycle` isn't available. - - A Meshtastic device is on a non-PPPS hub → note it; moving to a PPPS hub unlocks auto-recovery. - - `uhubctl_list` raises `ConfigError: uhubctl not found` → report as "uhubctl not installed"; don't treat as a device fault. - -5. **Render per-device report** as a compact block: - - ```text - [nrf52 @ /dev/cu.usbmodem1101] fw=2.7.23.bce2825, hw=RAK4631 - owner : Meshtastic 40eb / 40eb - region/band : US, channel 88, LONG_FAST - tx_power : 30 dBm, hop_limit=3 - peers : 1 (esp32s3 0x433c2428, pubkey ✓, SNR 6.0 / RSSI -24 dBm) - primary ch : McpTest - hub : 1-1.3 port 2 (PPPS, uhubctl-controllable) - firmware : no panics in last 3s - ``` - - Flag abnormalities inline with `⚠︎ ` — missing pubkey on a known peer, region UNSET, mismatched channel name, device on non-PPPS hub, etc. - -6. **Cross-device correlation** (when >1 device selected): - - Do both see each other in `nodesByNum`? - - Do `region`, `channel_num`, `modem_preset` match across devices? - - Do the primary channel names match? (Different name → different PSK → no decode.) - -7. **Suggest next steps only for recognizable failure modes**, never speculatively: - - Stale PKI one-way → "`/mcp-test tests/mesh/test_direct_with_ack.py` — the test's retry+nodeinfo-ping heals this." - - Region mismatch → "re-bake one side via `./mcp-server/run-tests.sh --force-bake`." - - Device unreachable, DFU reachable → `touch_1200bps(port=...)` + `pio_flash`. If not even DFU responds and the device is on a PPPS hub, escalate to `uhubctl_cycle(role=..., confirm=True)`. - - CP2102-wedged-driver on macOS → see `run-tests.sh` notes. - -## Hard constraints - -- **Read-only.** No `set_config`, no `reboot`, no `factory_reset`, no `flash`. If the operator wants mutation, they'll escalate explicitly. -- **Open/query/close per device.** Never hold multiple SerialInterfaces to the same port. The port lock is exclusive. -- **Don't infer env beyond the VID map** — if the operator has an unusual board, ask them which env to use rather than guessing. diff --git a/.github/prompts/mcp-repro.prompt.md b/.github/prompts/mcp-repro.prompt.md deleted file mode 100644 index 3a7c5c3de99..00000000000 --- a/.github/prompts/mcp-repro.prompt.md +++ /dev/null @@ -1,68 +0,0 @@ ---- -mode: agent -description: Re-run a specific test N times to triage flakes; diff firmware logs between passes and failures (Copilot equivalent of the Claude Code /repro slash command) ---- - -# `/mcp-repro` — flakiness triage for one test - -Equivalent of `.claude/commands/repro.md`. Use when the operator says "that one test is flaky — dig in", "repro the direct_with_ack failure", "why does X sometimes fail?". - -## What to do - -1. **Parse the operator's input** into two pieces: - - **Test identifier** — either a pytest node id (has `::` or starts with `tests/`) or a `-k`-style filter (plain substring like `direct_with_ack`). - - **Count** — integer, default `5`, cap at `20`. If the operator asks for 50, negotiate down and explain (airtime + USB wear). - -2. **Sanity-check the hub** via the `list_devices` MCP tool. If the test name references `nrf52` or `esp32s3` and the matching VID isn't present, stop and report — re-running won't help. - -3. **Loop** N times. Each iteration: - - ```bash - ./mcp-server/run-tests.sh --tb=short -p no:cacheprovider - ``` - - `-p no:cacheprovider` keeps pytest from caching anything between iterations. Capture: exit code, duration, and (on failure) the `Meshtastic debug` firmware-log section from `mcp-server/tests/report.html`. - -4. **Tally** results as you go: - - ```text - attempt 1: PASS (42s) - attempt 2: FAIL (128s) ← fw log captured - attempt 3: PASS (39s) - attempt 4: FAIL (121s) - attempt 5: PASS (41s) - -------------------------------------------------- - pass rate: 3/5 (60%) | mean duration: 74s - ``` - -5. **On mixed outcomes, diff the firmware logs** between one representative pass and one representative fail. Focus on: - - Error-level lines present only in failures (`PKI_UNKNOWN_PUBKEY`, `Alloc an err=`, `Skip send`, `No suitable channel`, `NAK`) - - Timing around the assertion point (broadcast sent? ACK received? retry fired?) - - Device-state fields that changed between attempts - - Surface the top 3 differences as a compact "passes when / fails when" table with uptime timestamps. Don't dump full logs. - -6. **Classify** the flake into one of: - - **LoRa airtime collision** — pass rate improves with fewer concurrent transmitters. Suggest a `time.sleep` gap or retry bump in the test body. - - **PKI key staleness** — first attempt fails, subsequent ones pass; existing retry-loop pattern in `test_direct_with_ack.py` is the fix. - - **NodeInfo cooldown** — `Skip send NodeInfo since we sent it <600s ago` in fail-only logs; needs a `broadcast_nodeinfo_ping()` warmup. - - **Hardware-specific** — one direction consistently fails, firmware versions differ, CP2102 driver wedged, etc. For a device wedged past `touch_1200bps`, recommend `uhubctl_cycle(role=..., confirm=True)` to hard-power-cycle its hub port (requires `uhubctl` installed). - - **Device went dark mid-run** — fails from some iteration onward and never recovers; firmware log stops arriving. Almost always a Guru crash with frozen CDC. Recommend `uhubctl_cycle` before the next iteration; escalate to replug if that also fails. - - **Unknown** — say so. Don't invent a root cause. - -7. **Report back** with: - - Pass rate + mean duration. - - Classification + the specific log evidence for it. - - A concrete next step (tighter assertion, more retries, open `/mcp-diagnose`, file a bug, nothing). - -## Examples - -- `tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[esp32s3->nrf52] 10` — 10 runs of that parametrized case. -- `broadcast_delivers` — no `::`, no `tests/`; treat as `-k broadcast_delivers`; runs every match 5 times. -- `tests/telemetry/test_device_telemetry_broadcast.py 3` — shorter count for a slow test. - -## Notes - -- If the FIRST attempt fails and the rest pass, that's a state-leak signature — suggest starting from `--force-bake` or a clean device state rather than chasing the first-failure firmware logs. -- If ALL N fail, this isn't a flake — it's a regression. Say so, stop iterating, escalate to `/mcp-test` for full-suite context. -- Don't rebuild firmware during triage. Flakes that only reproduce under different firmware belong in a separate session with a plan. diff --git a/.github/prompts/mcp-test.prompt.md b/.github/prompts/mcp-test.prompt.md deleted file mode 100644 index 148569e83da..00000000000 --- a/.github/prompts/mcp-test.prompt.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -mode: agent -description: Run the mcp-server test suite and interpret results (Copilot equivalent of the Claude Code /test slash command) ---- - -# `/mcp-test` — mcp-server test runner with interpretation - -Equivalent of the Claude Code `/test` slash command in `.claude/commands/test.md`. Use this when the operator asks you to "run the tests", "check the mcp test suite", "run the mesh tests", etc. - -## What to do - -1. **Invoke the wrapper** from the firmware repo root: - - ```bash - ./mcp-server/run-tests.sh [pytest-args] - ``` - - If the operator specified a subset (e.g. "just the mesh tests"), pass it through as `tests/mesh` or a pytest `-k filter`. If they said nothing, use the wrapper's defaults (full suite with pytest-html report). - - The wrapper auto-detects connected Meshtastic devices, maps each to its PlatformIO env, exports the required env vars, and invokes pytest. Zero pre-flight config needed from the operator. - -2. **Read the pre-flight header** (first few lines of wrapper output). The `detected hub :` line lists role → port → env mappings. If it reads `(none)`, the wrapper narrowed to `tests/unit` only — call that out explicitly so the operator knows hardware tiers were skipped. - -3. **On pass**: one-line summary like `N passed, M skipped in `. Don't enumerate test names. DO mention any non-placeholder SKIPs and name the cause: - - `"role not present on hub"` → device unplugged; operator should reconnect. - - `"firmware not baked with USERPREFS_UI_TEST_LOG"` → tests/ui skipped; the UI-log compile macro isn't in the baked firmware. Suggest `--force-bake`. - - `"uhubctl not installed"` → tests/recovery + `test_peer_offline_recovery` skipped. Suggest `brew install uhubctl` / `apt install uhubctl`. - - `"no PPPS-capable hubs detected"` → tests/recovery skipped because the attached hub doesn't support per-port power switching; won't run on that setup. - - `"opencv-python-headless is not installed"` → tests/ui auto-deselected by `run-tests.sh`. Suggest `pip install -e 'mcp-server/.[ui]'`. - -4. **On failure**: open `mcp-server/tests/report.html` (pytest-html output, self-contained) and extract the `Meshtastic debug` section for each failed test. That section includes a firmware log stream (last 200 lines) and device state dump. For each failure, summarise: - - test name - - one-line assertion message - - the specific firmware log lines that explain why (look for `PKI_UNKNOWN_PUBKEY`, `Skip send NodeInfo`, `Error=`, `Guru Meditation`, `assertion failed`, `No suitable channel`) - - for UI-tier failures also check `mcp-server/tests/ui_captures///transcript.md` (per-step frame + OCR) - -5. **Classify each failure** as one of: - - **Transient flake** — LoRa collision, first-attempt NAK with self-heal pattern, timing-sensitive assertion. Suggest `/mcp-repro ` to confirm. - - **Environmental** — device unreachable, port busy, CP2102 driver wedged on macOS. Suggest recovery in escalation order: (a) replug USB, (b) `touch_1200bps` + `pio_flash` for nRF52 DFU, (c) `uhubctl_cycle(role=..., confirm=True)` for a device wedged past DFU (needs `uhubctl` installed; `baked_single` does this once automatically when available). Also check `git status userPrefs.jsonc`. - - **Regression** — same assertion fails repeatedly on re-runs, firmware log shows novel errors. Identify the firmware module likely responsible. - -6. **Do NOT run destructive recovery automatically**. If a failure looks like it needs a reflash, factory*reset, `uhubctl_cycle`, or replug — \_describe the steps* and let the operator decide. Never burn airtime or flash cycles without approval. - -## Arguments convention - -Operators generally invoke this prompt either with no arguments (full suite) or with a specific subset. Examples: - -- `tests/mesh` — one tier -- `tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip` — one test -- `--force-bake` — reflash devices first -- `-k telemetry` — name-filter - -## Side-effects to confirm in your summary - -- `userPrefs.jsonc` should be clean after a successful run. The session fixture in `mcp-server/tests/conftest.py` (`_session_userprefs`) snapshots and restores. Check `git status --porcelain userPrefs.jsonc` and report if it's non-empty. -- `mcp-server/tests/report.html` and `junit.xml` regenerate on every run. -- The wrapper prints a warning if a `.mcp-session-bak` sidecar was left over from a crashed prior session and auto-restores from it — mention that if it happened. diff --git a/.github/prompts/new-module.prompt.md b/.github/prompts/new-module.prompt.md index 08b2395970a..9c6635aa801 100644 --- a/.github/prompts/new-module.prompt.md +++ b/.github/prompts/new-module.prompt.md @@ -6,9 +6,9 @@ Guide for developing a new Meshtastic firmware module. Choose the appropriate base class: -1. **`MeshModule`** — Raw base class. Override `wantPacket()` and `handleReceived()`. Returns `ProcessMessage::STOP` or `ProcessMessage::CONTINUE`. -2. **`SinglePortModule`** — Handles a single `meshtastic_PortNum`. Constructor takes `(name, portNum)`. Simplified `wantPacket()` checking `decoded.portnum`. Use `allocDataPacket()` to create outgoing packets. -3. **`ProtobufModule`** — Template for protobuf-encoded modules. Constructor takes `(name, portNum, fields)`. Override `handleReceivedProtobuf()`. Use `allocDataProtobuf(payload)` to create outgoing packets. +1. **`MeshModule`** - Raw base class. Override `wantPacket()` and `handleReceived()`. Returns `ProcessMessage::STOP` or `ProcessMessage::CONTINUE`. +2. **`SinglePortModule`** - Handles a single `meshtastic_PortNum`. Constructor takes `(name, portNum)`. Simplified `wantPacket()` checking `decoded.portnum`. Use `allocDataPacket()` to create outgoing packets. +3. **`ProtobufModule`** - Template for protobuf-encoded modules. Constructor takes `(name, portNum, fields)`. Override `handleReceivedProtobuf()`. Use `allocDataProtobuf(payload)` to create outgoing packets. Most modules also mix in `concurrency::OSThread` for periodic background tasks. @@ -32,7 +32,7 @@ class MyModule : public ProtobufModule, private concurrenc // Generate response packet (optional) virtual meshtastic_MeshPacket *allocReply() override; - // Periodic task — return next run interval in ms, or disable() + // Periodic task - return next run interval in ms, or disable() virtual int32_t runOnce() override; // Modify packet in-flight before delivery (optional) diff --git a/.github/prompts/new-sensor.prompt.md b/.github/prompts/new-sensor.prompt.md index e02fc246203..67eaaba81d5 100644 --- a/.github/prompts/new-sensor.prompt.md +++ b/.github/prompts/new-sensor.prompt.md @@ -6,10 +6,10 @@ Guide for adding a new I2C telemetry sensor driver to Meshtastic firmware. Telemetry sensors live in `src/modules/Telemetry/Sensor/`. There are 50+ existing drivers organized by measurement type. Each sensor integrates with one of the telemetry modules: -- **EnvironmentTelemetryModule** — Temperature, humidity, pressure, gas, light -- **AirQualityTelemetryModule** — Particulate matter, VOCs -- **PowerTelemetryModule** — Voltage, current, power monitoring -- **HealthTelemetryModule** — Heart rate, SpO2, body temperature +- **EnvironmentTelemetryModule** - Temperature, humidity, pressure, gas, light +- **AirQualityTelemetryModule** - Particulate matter, VOCs +- **PowerTelemetryModule** - Voltage, current, power monitoring +- **HealthTelemetryModule** - Heart rate, SpO2, body temperature ## Sensor Driver Pattern @@ -75,10 +75,10 @@ The scan runs at boot and populates a device map that telemetry modules use to d If the sensor provides data not covered by existing telemetry fields: 1. Add fields to the appropriate message in `protobufs/meshtastic/telemetry.proto`: - - `EnvironmentMetrics` — Environmental measurements - - `AirQualityMetrics` — Air quality data - - `PowerMetrics` — Power/energy data - - `HealthMetrics` — Health/biometric data + - `EnvironmentMetrics` - Environmental measurements + - `AirQualityMetrics` - Air quality data + - `PowerMetrics` - Power/energy data + - `HealthMetrics` - Health/biometric data 2. Add a `.options` constraint if needed (field sizes for nanopb) 3. Regenerate: `bin/regen-protos.sh` diff --git a/.github/prompts/new-variant.prompt.md b/.github/prompts/new-variant.prompt.md index 666e264e0bd..a5e401951ba 100644 --- a/.github/prompts/new-variant.prompt.md +++ b/.github/prompts/new-variant.prompt.md @@ -20,14 +20,14 @@ variants/ Each variant needs at minimum: -- `variant.h` — Pin definitions and hardware capabilities -- `platformio.ini` — Build configuration +- `variant.h` - Pin definitions and hardware capabilities +- `platformio.ini` - Build configuration Optional files: -- `pins_arduino.h` — Arduino pin mapping overrides -- `rfswitch.h` — RF switch control for multi-band radios -- `nicheGraphics.h` — InkHUD e-ink configuration +- `pins_arduino.h` - Arduino pin mapping overrides +- `rfswitch.h` - RF switch control for multi-band radios +- `nicheGraphics.h` - InkHUD e-ink configuration ## variant.h Template @@ -101,25 +101,25 @@ upload_speed = 921600 ### Common Base Configs -- `esp32_base` / `esp32-common.ini` — ESP32 -- `esp32s3_base` — ESP32-S3 -- `esp32c3_base` — ESP32-C3 -- `esp32c6_base` — ESP32-C6 -- `nrf52840_base` / `nrf52.ini` — nRF52840 -- `rp2040_base` — RP2040/RP2350 +- `esp32_base` / `esp32-common.ini` - ESP32 +- `esp32s3_base` - ESP32-S3 +- `esp32c3_base` - ESP32-C3 +- `esp32c6_base` - ESP32-C6 +- `nrf52840_base` / `nrf52.ini` - nRF52840 +- `rp2040_base` - RP2040/RP2350 ### Support Levels -- `custom_meshtastic_support_level = 1` — Built on every PR (actively supported) -- `custom_meshtastic_support_level = 2` — Built only on merge to main branches -- `board_level = extra` — Only built on full releases +- `custom_meshtastic_support_level = 1` - Built on every PR (actively supported) +- `custom_meshtastic_support_level = 2` - Built only on merge to main branches +- `board_level = extra` - Only built on full releases ## Build Manifest Metadata `bin/platformio-custom.py` emits UI capability flags in the build manifest: -- `custom_meshtastic_has_mui = true/false` — Override MUI detection -- `custom_meshtastic_has_ink_hud = true/false` — Override InkHUD detection +- `custom_meshtastic_has_mui = true/false` - Override MUI detection +- `custom_meshtastic_has_ink_hud = true/false` - Override InkHUD detection - Architecture names are normalized (e.g., `esp32s3` → `esp32-s3`) ## InkHUD E-Ink Variants @@ -127,7 +127,7 @@ upload_speed = 921600 For e-ink display variants using the InkHUD framework, add `nicheGraphics.h`: ```cpp -// nicheGraphics.h — InkHUD configuration for this variant +// nicheGraphics.h - InkHUD configuration for this variant #define INKHUD // Enable InkHUD // Configure display, applets, and refresh behavior per device ``` diff --git a/.github/workflows/build_debian_src.yml b/.github/workflows/build_debian_src.yml index 8d2076b113f..066727cff72 100644 --- a/.github/workflows/build_debian_src.yml +++ b/.github/workflows/build_debian_src.yml @@ -23,7 +23,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive path: meshtasticd diff --git a/.github/workflows/build_firmware.yml b/.github/workflows/build_firmware.yml index 47010468849..bafa63d3ea9 100644 --- a/.github/workflows/build_firmware.yml +++ b/.github/workflows/build_firmware.yml @@ -23,7 +23,7 @@ jobs: outputs: artifact-id: ${{ steps.upload-firmware.outputs.artifact-id }} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/build_macos_bin.yml b/.github/workflows/build_macos_bin.yml index d0e89d7da6e..ccd64931885 100644 --- a/.github/workflows/build_macos_bin.yml +++ b/.github/workflows/build_macos_bin.yml @@ -16,7 +16,7 @@ jobs: runs-on: macos-${{ inputs.macos_ver }} steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/build_one_target.yml b/.github/workflows/build_one_target.yml index 706b9cfe79b..b6ec92edb75 100644 --- a/.github/workflows/build_one_target.yml +++ b/.github/workflows/build_one_target.yml @@ -4,9 +4,14 @@ on: workflow_dispatch: inputs: # trunk-ignore(checkov/CKV_GHA_7) + target: + type: string + required: false + description: Choose the target board, e.g. nrf52_promicro_diy_tcxo. If blank, will find available targets. arch: type: choice options: + - all - esp32 - esp32s3 - esp32c3 @@ -15,35 +20,21 @@ on: - rp2040 - rp2350 - stm32 - target: - type: string - required: false - description: Choose the target board, e.g. nrf52_promicro_diy_tcxo. If blank, will find available targets. - # find-target: - # type: boolean - # default: true - # description: 'Find the available targets' + description: Choose an arch to limit the search, or 'all' to search all architectures. + default: all permissions: read-all jobs: find-targets: - if: ${{ inputs.target == '' }} strategy: fail-fast: false matrix: arch: - - esp32 - - esp32s3 - - esp32c3 - - esp32c6 - - nrf52840 - - rp2040 - - rp2350 - - stm32 + - all runs-on: ubuntu-24.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - uses: actions/setup-python@v6 with: python-version: 3.x @@ -51,20 +42,43 @@ jobs: - run: pip install -U platformio - name: Generate matrix id: jsonStep + env: + BUILDTARGET: ${{ inputs.target }} + MATRIXARCH: ${{ inputs.arch }} run: | TARGETS=$(./bin/generate_ci_matrix.py ${{matrix.arch}} --level extra) - echo "Name: $GITHUB_REF_NAME" >> $GITHUB_STEP_SUMMARY - echo "Base: $GITHUB_BASE_REF" >> $GITHUB_STEP_SUMMARY - echo "Arch: ${{matrix.arch}}" >> $GITHUB_STEP_SUMMARY - echo "Ref: $GITHUB_REF" >> $GITHUB_STEP_SUMMARY - echo "Targets:" >> $GITHUB_STEP_SUMMARY - echo $TARGETS | jq -r 'sort_by(.board) |.[] | "- " + .board' >> $GITHUB_STEP_SUMMARY + if [ "$BUILDTARGET" = "" ]; then + echo "Name: $GITHUB_REF_NAME" >> $GITHUB_STEP_SUMMARY + echo "Base: $GITHUB_BASE_REF" >> $GITHUB_STEP_SUMMARY + echo "Arch: $MATRIXARCH" >> $GITHUB_STEP_SUMMARY + echo "Ref: $GITHUB_REF" >> $GITHUB_STEP_SUMMARY + echo "## 🎯 The following target boards are available to build:" >> $GITHUB_STEP_SUMMARY + echo "| Platform | Board |" >> $GITHUB_STEP_SUMMARY + echo "| -------- | ----- |" >> $GITHUB_STEP_SUMMARY + echo $TARGETS | jq -r 'sort_by(.board) | sort_by(.platform) |.[] | "| " + .platform + " | " + .board + " |" ' >> $GITHUB_STEP_SUMMARY + else + echo "We build this one:" >> $GITHUB_STEP_SUMMARY + ARCH=$(echo "$TARGETS" | jq --arg BUILDTARGET "$BUILDTARGET" -r '.[] | select(.board==$BUILDTARGET) | .platform') + echo "| Platform | Board |" >> $GITHUB_STEP_SUMMARY + echo "| -------- | ----- |" >> $GITHUB_STEP_SUMMARY + echo "| $ARCH | "$BUILDTARGET" |" >> $GITHUB_STEP_SUMMARY + echo "" >> $GITHUB_STEP_SUMMARY + if [[ "$ARCH" == "" ]]; then + echo "## ❌ Error: Target "$BUILDTARGET" not found!" >> $GITHUB_STEP_SUMMARY + else + echo "## ✅ Target "$BUILDTARGET" found, proceeding to build." >> $GITHUB_STEP_SUMMARY + fi + echo "You may need to refresh this page to make the built firmware appear below." >> $GITHUB_STEP_SUMMARY + echo "arch=$ARCH" >> $GITHUB_OUTPUT + fi + outputs: + arch: ${{ steps.jsonStep.outputs.arch }} version: if: ${{ inputs.target != '' }} runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Get release version string run: | echo "long=$(./bin/buildinfo.py long)" >> $GITHUB_OUTPUT @@ -78,12 +92,12 @@ jobs: build: if: ${{ inputs.target != '' && inputs.arch != 'native' }} - needs: [version] + needs: [version, find-targets] uses: ./.github/workflows/build_firmware.yml with: version: ${{ needs.version.outputs.long }} pio_env: ${{ inputs.target }} - platform: ${{ inputs.arch }} + platform: ${{ needs.find-targets.outputs.arch }} gather-artifacts: permissions: @@ -93,7 +107,7 @@ jobs: needs: [version, build] steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: ref: ${{github.event.pull_request.head.ref}} repository: ${{github.event.pull_request.head.repo.full_name}} diff --git a/.github/workflows/build_portduino_wasm.yml b/.github/workflows/build_portduino_wasm.yml new file mode 100644 index 00000000000..c6c32583a56 --- /dev/null +++ b/.github/workflows/build_portduino_wasm.yml @@ -0,0 +1,61 @@ +name: Build PortDuino WASM + +# Reusable workflow - called as the `build-wasm` job of the main CI workflow +# (main_matrix.yml), so the WebAssembly portduino node is built like every other +# platform. Builds the [env:native-wasm] target (src/platform/portduino/wasm/) with +# `pio run -e native-wasm` so it can't silently bit-rot. Software/CI only - asserts the +# full mesh stack + RadioLib + Crypto compile and link to meshnode.{mjs,wasm} +# via the meshtastic/platform-wasm PlatformIO platform (emcc/Asyncify). It does +# NOT exercise the radio (that's CH341/WebUSB, hardware). + +on: + workflow_call: + workflow_dispatch: + +permissions: {} + +jobs: + build-wasm: + name: Build PortDuino WASM + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v7 + with: + submodules: recursive + + # Python + PlatformIO. (`pio run -e native-wasm` installs its own libdeps and the + # platform-wasm/framework-portduino packages; it needs no native apt libs.) + - name: Setup native build + uses: ./.github/actions/setup-native + + - name: Setup Emscripten SDK + uses: emscripten-core/setup-emsdk@v16 + with: + version: 6.0.1 + actions-cache-folder: emsdk-cache + + - name: Build PortDuino WASM + run: platformio run -e native-wasm + + - name: Assert WASM artifacts + run: | + OUT=.pio/build/native-wasm + if [ ! -s "$OUT/meshnode.mjs" ] || [ ! -s "$OUT/meshnode.wasm" ]; then + echo "::error::wasm artifacts missing"; exit 1 + fi + # The emcc link succeeds even if embedded EM_ASM JS is malformed, so + # parse the generated ES-module loader to catch runtime-JS breakage + # (e.g. a formatter splitting !== inside EM_ASM). + node --check "$OUT/meshnode.mjs" || { echo "::error::meshnode.mjs failed JS parse"; exit 1; } + ls -lah "$OUT"/meshnode.* + + - name: Upload WASM artifacts + if: success() + uses: actions/upload-artifact@v7 + with: + name: portduino-wasm + overwrite: true + path: | + .pio/build/native-wasm/meshnode.mjs + .pio/build/native-wasm/meshnode.wasm + retention-days: 14 diff --git a/.github/workflows/docker_build.yml b/.github/workflows/docker_build.yml index 8a3ef0e6cd7..03a1f91cd4c 100644 --- a/.github/workflows/docker_build.yml +++ b/.github/workflows/docker_build.yml @@ -47,7 +47,7 @@ jobs: runs-on: ${{ inputs.runs-on }} steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/docker_manifest.yml b/.github/workflows/docker_manifest.yml index 4bfdfe37e47..92c71bd852b 100644 --- a/.github/workflows/docker_manifest.yml +++ b/.github/workflows/docker_manifest.yml @@ -103,7 +103,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/firmware-size-comment.yml b/.github/workflows/firmware-size-comment.yml new file mode 100644 index 00000000000..4d7f5c3fcbf --- /dev/null +++ b/.github/workflows/firmware-size-comment.yml @@ -0,0 +1,62 @@ +name: Post Firmware Size Comment + +on: + workflow_run: + workflows: [CI] + types: [completed] + +permissions: + pull-requests: write + actions: read + +jobs: + post-size-comment: + if: > + github.event.workflow_run.event == 'pull_request' && + github.event.workflow_run.conclusion != 'cancelled' && + github.repository == 'meshtastic/firmware' + continue-on-error: true + runs-on: ubuntu-latest + steps: + - name: Download size report + id: download + uses: actions/download-artifact@v8 + continue-on-error: true + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ github.event.workflow_run.id }} + name: size-report + path: ./ + + - name: Post or update PR comment + if: steps.download.outcome == 'success' + uses: actions/github-script@v9 + with: + script: | + const fs = require('fs'); + const marker = ''; + const body = fs.readFileSync('./size-report.md', 'utf8'); + const prNumber = parseInt(fs.readFileSync('./pr-number.txt', 'utf8').trim(), 10); + + const { data: comments } = await github.rest.issues.listComments({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + }); + + const existing = comments.find(c => c.body.includes(marker)); + if (existing) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: existing.id, + body, + }); + } else { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: prNumber, + body, + }); + } diff --git a/.github/workflows/flasher-link-comment.yml b/.github/workflows/flasher-link-comment.yml new file mode 100644 index 00000000000..e71b8bb50a2 --- /dev/null +++ b/.github/workflows/flasher-link-comment.yml @@ -0,0 +1,187 @@ +name: Post Web Flasher Link Comment + +on: + workflow_run: + workflows: [CI] + types: [completed] + +permissions: + pull-requests: write + actions: read + +jobs: + post-flasher-link: + if: > + github.event.workflow_run.event == 'pull_request' && + github.event.workflow_run.conclusion != 'cancelled' && + github.repository == 'meshtastic/firmware' + continue-on-error: true + runs-on: ubuntu-latest + steps: + # Per-board manifests carry the firmware's own metadata (activelySupported, + # displayName, ...) generated from each target's custom_meshtastic_* config. + - name: Download board manifests + uses: actions/download-artifact@v8 + continue-on-error: true + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + run-id: ${{ github.event.workflow_run.id }} + pattern: manifest-* + path: ./manifests + merge-multiple: true + + - name: Post or update web flasher link comment + uses: actions/github-script@v9 + with: + script: | + const marker = ''; + const run = context.payload.workflow_run; + const { owner, repo } = context.repo; + + // Resolve the PR by matching the run's head SHA against the repo's open + // PRs. workflow_run.pull_requests is empty for fork PRs, and + // listPullRequestsAssociatedWithCommit won't return an open fork PR by + // its head commit - but pulls.list includes fork PRs. Matching on head + // SHA also enforces that the run is for the PR's current commit, so stale + // re-runs of an outdated commit won't match. + const openPrs = await github.paginate(github.rest.pulls.list, { + owner, repo, state: 'open', per_page: 100, + }); + const pr = openPrs.find((p) => p.head.sha === run.head_sha); + if (!pr) { + core.info(`No open pull request matches commit ${run.head_sha}; skipping.`); + return; + } + const prNumber = pr.number; + + // Restrict to trusted authors. NOTE: author_association is computed for + // the GITHUB_TOKEN, which cannot see *private/concealed* org memberships - + // those members come back as CONTRIBUTOR, not MEMBER. So gating on MEMBER + // alone silently excludes most maintainers. We allow the trusted set the + // token can actually identify (members, collaborators, and anyone with a + // previously merged PR). For strict members-only you'd need an org-read + // App/PAT token to call orgs.checkMembershipForUser. + const allowedAssociations = ['OWNER', 'MEMBER', 'COLLABORATOR', 'CONTRIBUTOR']; + if (!allowedAssociations.includes(pr.author_association)) { + core.info(`Author association ${pr.author_association} is not trusted; skipping.`); + return; + } + + // Require at least one per-arch firmware artifact from gather-artifacts + const artifacts = await github.paginate(github.rest.actions.listWorkflowRunArtifacts, { + owner, repo, run_id: run.id, per_page: 100, + }); + const archRe = /^firmware-(esp32|esp32s3|esp32c3|esp32c6|nrf52840|rp2040|rp2350|stm32)-(\d+\.\d+\.\d+\.[0-9a-f]+)$/; + const archArtifacts = artifacts.filter((a) => archRe.test(a.name) && !a.expired); + if (archArtifacts.length === 0) { + core.info('No per-arch firmware artifacts found; skipping.'); + return; + } + + const version = archRe.exec(archArtifacts[0].name)[2]; + const expiresAt = archArtifacts[0].expires_at + ? new Date(archArtifacts[0].expires_at).toISOString().slice(0, 10) + : null; + + // Read each built board's manifest (.mt.json). activelySupported, + // displayName and architecture come straight from the board's + // custom_meshtastic_* platformio config, so the list is in sync with + // the firmware itself - no external device database needed. + const fs = require('fs'); + let boards = []; + try { + boards = fs.readdirSync('./manifests') + .filter((f) => f.endsWith('.mt.json')) + .map((f) => { + try { return JSON.parse(fs.readFileSync(`./manifests/${f}`, 'utf8')); } + catch { return null; } + }) + .filter((m) => m && m.activelySupported === true && m.platformioTarget) + .map((m) => ({ + board: m.platformioTarget, + platform: m.architecture || '', + // displayName is maintainer-authored text; escape table-breaking pipes + displayName: String(m.displayName || m.platformioTarget).replace(/\|/g, '\\|'), + image: Array.isArray(m.images) && m.images[0] ? String(m.images[0]) : '', + })) + .sort((a, b) => a.board.localeCompare(b.board)); + } catch (e) { + core.warning(`Could not read board manifests: ${e.message}`); + } + + const flasherUrl = `https://flasher.meshtastic.org/?pr=${prNumber}`; + // Device illustrations are served by the flasher from the same image + // names the manifest declares (custom_meshtastic_images). The flasher + // serves its SPA shell (HTML, 200) for unknown paths, so confirm each + // image really resolves to an image before linking it. + const imageBase = 'https://flasher.meshtastic.org/img/devices/'; + await Promise.all(boards.map(async (b) => { + if (!b.image) return; + try { + const res = await fetch(`${imageBase}${encodeURIComponent(b.image)}`); + const type = res.headers.get('content-type') || ''; + if (!res.ok || !type.startsWith('image/')) b.image = ''; + } catch { b.image = ''; } + })); + + const boardLines = boards + .map((b) => { + const img = b.image ? `` : ''; + return `| ${img} | ${b.displayName} | [\`${b.board}\`](${flasherUrl}&device=${encodeURIComponent(b.board)}) | ${b.platform} |`; + }) + .join('\n'); + + // Shields.io badges. Only non-user-controlled, charset-constrained values + // (version, commit sha, counts, dates) go into badge URLs - never board + // names or the PR title - so the rendered comment cannot be spoofed. + const shieldText = (s) => + encodeURIComponent(String(s).replace(/-/g, '--').replace(/_/g, '__').replace(/ /g, '_')); + const shield = (label, message, color) => + `https://img.shields.io/badge/${shieldText(label)}-${shieldText(message)}-${color}`; + const buttonUrl = + `https://img.shields.io/badge/${shieldText('Flash this PR in the Web Flasher')}-2C2D3C?style=for-the-badge`; + const badges = [ + `![firmware](${shield('firmware', version, '67EA94')})`, + `![commit](${shield('commit', run.head_sha.slice(0, 7), '2C2D3C')})`, + `![boards](${shield('boards', boards.length, '5C6BC0')})`, + ]; + if (expiresAt) badges.push(`![expires](${shield('expires', expiresAt, '9A4E00')})`); + + // Only render the board table when there are supported boards to list + const boardTable = boards.length > 0 ? [ + `

Supported boards built by this PR (${boards.length})`, + '', + '| | Device | Board | Platform |', + '| --- | --- | --- | --- |', + boardLines, + '', + '
', + '', + ] : []; + + const body = [ + marker, + '## ⚡ Try this PR in the Web Flasher', + '', + `[![Flash this PR in the Web Flasher](${buttonUrl})](${flasherUrl})`, + '', + badges.join(' '), + '', + '> [!WARNING]', + '> This is an automated, unreviewed CI test build. Back up your device configuration', + '> before flashing, and only flash devices you are able to recover.', + '', + ...boardTable, + `*Build artifacts expire${expiresAt ? ` on ${expiresAt}` : ' after 30 days'}. Updated for \`${run.head_sha.slice(0, 7)}\`.*`, + ].join('\n'); + + // Sticky comment: update in place when the marker is found + const comments = await github.paginate(github.rest.issues.listComments, { + owner, repo, issue_number: prNumber, per_page: 100, + }); + const existing = comments.find((c) => c.body?.includes(marker)); + if (existing) { + await github.rest.issues.updateComment({ owner, repo, comment_id: existing.id, body }); + } else { + await github.rest.issues.createComment({ owner, repo, issue_number: prNumber, body }); + } diff --git a/.github/workflows/flasher-link-placeholder.yml b/.github/workflows/flasher-link-placeholder.yml new file mode 100644 index 00000000000..c239225af2d --- /dev/null +++ b/.github/workflows/flasher-link-placeholder.yml @@ -0,0 +1,62 @@ +name: Post Web Flasher Build Placeholder + +# Drops an immediate "build in progress" comment when a PR opens, so the web +# flasher entry shows up right away. The real CI-driven workflow +# (flasher-link-comment.yml) later replaces it in place via the shared marker. +# +# SECURITY: this uses pull_request_target (write token, runs for fork PRs) but is +# safe because it never checks out or runs PR code and posts a fully static body +# - no PR title, branch name, or other untrusted input is used anywhere. + +on: + pull_request_target: + types: [opened, reopened] + +permissions: + pull-requests: write + +jobs: + post-placeholder: + if: github.repository == 'meshtastic/firmware' + continue-on-error: true + runs-on: ubuntu-latest + steps: + - name: Post web flasher build-in-progress placeholder + uses: actions/github-script@v9 + with: + script: | + const marker = ''; + const { owner, repo } = context.repo; + const pr = context.payload.pull_request; + + // Trusted authors only (matches the real workflow). author_association + // can't reflect private org membership for the token, so concealed + // members appear as CONTRIBUTOR - include it, or maintainers are excluded. + const allowedAssociations = ['OWNER', 'MEMBER', 'COLLABORATOR', 'CONTRIBUTOR']; + if (!allowedAssociations.includes(pr.author_association)) { + core.info(`Author association ${pr.author_association} is not trusted; skipping.`); + return; + } + + // Only seed a placeholder when no flasher comment exists yet - never + // overwrite a real (or existing placeholder) comment. + const comments = await github.paginate(github.rest.issues.listComments, { + owner, repo, issue_number: pr.number, per_page: 100, + }); + if (comments.some((c) => c.body?.includes(marker))) { + core.info('Flasher comment already exists; nothing to do.'); + return; + } + + const body = [ + marker, + '## ⚡ Try this PR in the Web Flasher', + '', + '> [!NOTE]', + '> Building this pull request… the flash button, badges and supported-board', + '> list will appear here automatically once CI finishes.', + ].join('\n'); + + await github.rest.issues.createComment({ + owner, repo, issue_number: pr.number, body, + }); diff --git a/.github/workflows/hook_copr.yml b/.github/workflows/hook_copr.yml index c419848a863..aba7e9c0dd0 100644 --- a/.github/workflows/hook_copr.yml +++ b/.github/workflows/hook_copr.yml @@ -18,7 +18,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/main_matrix.yml b/.github/workflows/main_matrix.yml index 3505d950e35..a0097697318 100644 --- a/.github/workflows/main_matrix.yml +++ b/.github/workflows/main_matrix.yml @@ -26,7 +26,19 @@ on: - "**.md" #- "**.yml" + schedule: + # Nightly develop build, published to meshtastic.github.io firmware-nightly/ (no GitHub release). + # Scheduled runs execute on the default branch (develop). 09:00 UTC avoids the 00:00 tests + # and 02:00 daily_packaging crons. + - cron: 0 9 * * * # Nightly develop build/publish (default branch is develop) + workflow_dispatch: + inputs: + # trunk-ignore(checkov/CKV_GHA_7): intentional manual-test switch for the nightly publish path + nightly: + description: "Nightly mode: build + publish develop to github.io firmware-nightly/ (skips creating a GitHub release)" + type: boolean + default: false permissions: read-all @@ -40,7 +52,7 @@ jobs: - check runs-on: ubuntu-24.04 steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - uses: actions/setup-python@v6 with: python-version: 3.x @@ -64,7 +76,7 @@ jobs: version: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Get release version string run: | echo "long=$(./bin/buildinfo.py long)" >> $GITHUB_OUTPUT @@ -82,11 +94,12 @@ jobs: fail-fast: false matrix: check: ${{ fromJson(needs.setup.outputs.check) }} - # Use 'arctastic' self-hosted runner pool when checking in the main repo - runs-on: ${{ github.repository_owner == 'meshtastic' && 'arctastic' || 'ubuntu-latest' }} - if: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'meshtastic/firmware' }} + # Runs on GitHub-hosted runners so checks don't compete with builds for the + # self-hosted 'arctastic' pool (which builds use). + runs-on: ubuntu-latest + if: ${{ github.event_name != 'workflow_dispatch' && github.repository == 'meshtastic/firmware' && github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: submodules: recursive - name: Check ${{ matrix.check.board }} @@ -109,7 +122,7 @@ jobs: platform: ${{ matrix.build.platform }} build-debian-src: - if: github.repository == 'meshtastic/firmware' + if: ${{ github.repository == 'meshtastic/firmware' && github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} uses: ./.github/workflows/build_debian_src.yml with: series: UNRELEASED @@ -117,6 +130,7 @@ jobs: secrets: inherit MacOS: + if: ${{ github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} strategy: fail-fast: false matrix: @@ -138,14 +152,24 @@ jobs: secrets: inherit test-native: - if: ${{ !contains(github.ref_name, 'event/') && github.repository == 'meshtastic/firmware' }} + if: ${{ !contains(github.ref_name, 'event/') && github.repository == 'meshtastic/firmware' && github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} permissions: # Needed for dorny/test-reporter. contents: read actions: read checks: write uses: ./.github/workflows/test_native.yml + build-wasm: + if: ${{ github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} + # Build the WebAssembly portduino node ([env:native-wasm]) as part of normal CI, + # like the other platforms. It's a dedicated job (not a row in the `build` + # matrix) because its artifact is meshnode.{mjs,wasm} - not a flashable + # .bin/.uf2/.hex - and it needs the Emscripten SDK; board_level=extra keeps + # it out of generate_ci_matrix.py. + uses: ./.github/workflows/build_portduino_wasm.yml + docker: + if: ${{ github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} permissions: # Needed for pushing to GHCR. contents: read packages: write @@ -189,7 +213,7 @@ jobs: needs: [version, build] steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 - uses: actions/download-artifact@v8 with: @@ -245,53 +269,161 @@ jobs: path: ./*.elf retention-days: 30 - shame: - if: github.repository == 'meshtastic/firmware' + firmware-size-report: + if: ${{ github.repository == 'meshtastic/firmware' && github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} continue-on-error: true + permissions: + contents: read + actions: read runs-on: ubuntu-latest needs: [build] steps: - - uses: actions/checkout@v6 - if: github.event_name == 'pull_request' - with: - filter: blob:none # means we download all the git history but none of the commit (except ones with checkout like the head) - fetch-depth: 0 - - name: Download the current manifests + - uses: actions/checkout@v7 + + - name: Download current manifests uses: actions/download-artifact@v8 with: - path: ./manifests-new/ + path: ./manifests/ pattern: manifest-* merge-multiple: true - - name: Upload combined manifests for later commit and global stats crunching. + + - name: Collect current firmware sizes + run: python3 bin/collect_sizes.py ./manifests/ ./current-sizes.json + + - name: Upload size report artifact uses: actions/upload-artifact@v7 - id: upload-manifest with: - name: manifests-${{ github.sha }} + name: firmware-sizes-${{ github.sha }} overwrite: true - path: manifests-new/*.mt.json - - name: Find the merge base + path: ./current-sizes.json + retention-days: 90 + + - name: Download baseline sizes from develop + if: github.event_name == 'pull_request' + continue-on-error: true + id: baseline-develop + env: + GH_TOKEN: ${{ github.token }} + run: | + RUN_ID=$(gh run list -R "${{ github.repository }}" \ + --workflow CI --branch develop --status success \ + --limit 1 --json databaseId --jq '.[0].databaseId // empty') + if [ -n "$RUN_ID" ]; then + ARTIFACT_NAME=$(gh api "repos/${{ github.repository }}/actions/runs/${RUN_ID}/artifacts" \ + --jq '.artifacts[] | select(.name | startswith("firmware-sizes-")) | select(.expired == false) | .name' | head -1) + if [ -n "$ARTIFACT_NAME" ]; then + gh run download "$RUN_ID" -R "${{ github.repository }}" \ + --name "$ARTIFACT_NAME" --dir ./baseline-develop/ + cp "./baseline-develop/current-sizes.json" ./develop-sizes.json + echo "found=true" >> "$GITHUB_OUTPUT" + else + echo "found=false" >> "$GITHUB_OUTPUT" + fi + else + echo "found=false" >> "$GITHUB_OUTPUT" + fi + + - name: Download baseline sizes from master if: github.event_name == 'pull_request' - run: echo "MERGE_BASE=$(git merge-base "origin/$base" "$head")" >> $GITHUB_ENV + continue-on-error: true + id: baseline-master env: - base: ${{ github.base_ref }} - head: ${{ github.sha }} - # Currently broken (for-loop through EVERY artifact -- rate limiting) - # - name: Download the old manifests - # if: github.event_name == 'pull_request' - # run: gh run download -R "$repo" --name "manifests-$merge_base" --dir manifest-old/ - # env: - # GH_TOKEN: ${{ github.token }} - # merge_base: ${{ env.MERGE_BASE }} - # repo: ${{ github.repository }} - # - name: Do scan and post comment - # if: github.event_name == 'pull_request' - # run: python3 bin/shame.py ${{ github.event.pull_request.number }} manifests-old/ manifests-new/ + GH_TOKEN: ${{ github.token }} + run: | + RUN_ID=$(gh run list -R "${{ github.repository }}" \ + --workflow CI --branch master --status success \ + --limit 1 --json databaseId --jq '.[0].databaseId // empty') + if [ -n "$RUN_ID" ]; then + ARTIFACT_NAME=$(gh api "repos/${{ github.repository }}/actions/runs/${RUN_ID}/artifacts" \ + --jq '.artifacts[] | select(.name | startswith("firmware-sizes-")) | select(.expired == false) | .name' | head -1) + if [ -n "$ARTIFACT_NAME" ]; then + gh run download "$RUN_ID" -R "${{ github.repository }}" \ + --name "$ARTIFACT_NAME" --dir ./baseline-master/ + cp "./baseline-master/current-sizes.json" ./master-sizes.json + echo "found=true" >> "$GITHUB_OUTPUT" + else + echo "found=false" >> "$GITHUB_OUTPUT" + fi + else + echo "found=false" >> "$GITHUB_OUTPUT" + fi + + - name: Generate size comparison report + if: github.event_name == 'pull_request' + id: report + run: | + ARGS="./current-sizes.json --budgets bin/ram_budgets.json" + if [ -f ./develop-sizes.json ]; then + ARGS="$ARGS --baseline develop:./develop-sizes.json" + fi + if [ -f ./master-sizes.json ]; then + ARGS="$ARGS --baseline master:./master-sizes.json" + fi + REPORT=$(python3 bin/size_report.py $ARGS) + if [ -z "$REPORT" ]; then + echo "has_report=false" >> "$GITHUB_OUTPUT" + else + echo "has_report=true" >> "$GITHUB_OUTPUT" + { + echo '' + echo '# Firmware Size Report' + echo '' + echo "$REPORT" + echo '' + echo '---' + echo "*Updated for ${{ github.sha }}*" + } > ./size-report.md + cat ./size-report.md >> "$GITHUB_STEP_SUMMARY" + fi + + - name: Save PR number + if: github.event_name == 'pull_request' && steps.report.outputs.has_report == 'true' + run: echo "${{ github.event.pull_request.number }}" > ./pr-number.txt + + - name: Upload size report + if: github.event_name == 'pull_request' && steps.report.outputs.has_report == 'true' + uses: actions/upload-artifact@v7 + with: + name: size-report + path: | + ./size-report.md + ./pr-number.txt + retention-days: 5 + + # RAM/flash guardrails: fails CI when an env listed in bin/ram_budgets.json + # exceeds its static RAM (.data+.bss) or flash budget. Kept separate from + # firmware-size-report, which is informational and continue-on-error. + size-budget-gate: + if: ${{ github.event_name != 'schedule' && github.event.inputs.nightly != 'true' }} + permissions: + contents: read + actions: read + runs-on: ubuntu-latest + needs: [build] + steps: + - uses: actions/checkout@v7 + + # No continue-on-error / empty-dir fallback: the gate must fail closed when + # the data it enforces on cannot be fetched (size_report.py additionally + # fails on missing budgeted envs under --enforce-budgets). + - name: Download current manifests + uses: actions/download-artifact@v8 + with: + path: ./manifests/ + pattern: manifest-* + merge-multiple: true + + - name: Collect current firmware sizes + run: python3 bin/collect_sizes.py ./manifests/ ./current-sizes.json + + - name: Enforce RAM/flash budgets + run: python3 bin/size_report.py ./current-sizes.json --budgets bin/ram_budgets.json --enforce-budgets release-artifacts: permissions: # Needed for 'gh release upload'. contents: write runs-on: ubuntu-latest - if: ${{ github.event_name == 'workflow_dispatch' && github.repository == 'meshtastic/firmware' }} + if: ${{ github.event_name == 'workflow_dispatch' && github.repository == 'meshtastic/firmware' && github.event.inputs.nightly != 'true' }} outputs: upload_url: ${{ steps.create_release.outputs.upload_url }} needs: @@ -303,7 +435,7 @@ jobs: # - MacOS steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 0 @@ -333,6 +465,7 @@ jobs: prerelease: true name: Meshtastic Firmware ${{ needs.version.outputs.long }} Alpha tag_name: v${{ needs.version.outputs.long }} + target_commitish: ${{ github.sha }} body: ${{ steps.release_notes.outputs.notes }} - name: Download source deb @@ -399,11 +532,11 @@ jobs: - rp2350 - stm32 runs-on: ubuntu-latest - if: ${{ github.event_name == 'workflow_dispatch' && github.repository == 'meshtastic/firmware'}} + if: ${{ github.event_name == 'workflow_dispatch' && github.repository == 'meshtastic/firmware' && github.event.inputs.nightly != 'true' }} needs: [release-artifacts, version] steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Setup Python uses: actions/setup-python@v6 @@ -451,14 +584,14 @@ jobs: publish-firmware: runs-on: ubuntu-24.04 - if: ${{ github.event_name == 'workflow_dispatch' }} + if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.nightly != 'true' }} needs: [release-firmware, version] env: targets: |- esp32,esp32s3,esp32c3,esp32c6,nrf52840,rp2040,rp2350,stm32 steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 0 @@ -503,3 +636,78 @@ jobs: user_email: github-actions[bot]@users.noreply.github.com commit_message: ${{ needs.version.outputs.long }} enable_jekyll: true + + # Nightly publish: refresh the single, stable firmware-nightly/ folder on + # meshtastic.github.io with the current develop build. Runs on the cron schedule + # (or a manual nightly=true dispatch) and never creates a GitHub release. The + # folder's release_notes.md is maintained by hand and deliberately left untouched. + publish-nightly: + runs-on: ubuntu-24.04 + if: ${{ (github.event_name == 'schedule' || github.event.inputs.nightly == 'true') && github.repository == 'meshtastic/firmware' }} + needs: [setup, version, gather-artifacts] + env: + targets: |- + esp32,esp32s3,esp32c3,esp32c6,nrf52840,rp2040,rp2350,stm32 + steps: + - name: Get firmware artifacts + uses: actions/download-artifact@v8 + with: + pattern: firmware-{${{ env.targets }}}-${{ needs.version.outputs.long }} + merge-multiple: true + path: ./stage + + - name: Generate Release manifest + run: | + jq -n --arg ver "${{ needs.version.outputs.long }}" --argjson targets ${{ toJson(needs.setup.outputs.all) }} '{ + "version": $ver, + "targets": $targets + }' > ./stage/firmware-${{ needs.version.outputs.long }}.json + + - name: Generate nightly pointer + run: | + jq -n \ + --arg ver "${{ needs.version.outputs.long }}" \ + --arg sha "${{ github.sha }}" \ + '{version: $ver, id: ("v" + $ver), title: ("Meshtastic Firmware " + $ver + " Nightly"), commit: $sha}' \ + > ./stage/index.json + + - name: Preserve manually-maintained release notes + # firmware-nightly/release_notes.md is edited by hand. Carry the current + # copy into ./stage so the keep_files:false publish (which refreshes the + # folder and clears stale nightly binaries) does not drop it. Seed a + # placeholder on the first run (404); fail closed on any other error so a + # transient fetch failure never clobbers the notes. + run: | + set -euo pipefail + url=https://raw.githubusercontent.com/meshtastic/meshtastic.github.io/master/firmware-nightly/release_notes.md + code=$(curl -sSL -o ./stage/release_notes.md -w '%{http_code}' --retry 5 --retry-all-errors "$url" || echo 000) + if [ "$code" = "200" ]; then + echo "Preserved existing release_notes.md" + elif [ "$code" = "404" ]; then + echo "No existing release_notes.md; seeding placeholder" + printf '# Nightly (develop)\n\nAutomated nightly build from the `develop` branch. Edit these notes by hand.\n' > ./stage/release_notes.md + else + echo "Unexpected HTTP $code fetching release_notes.md; refusing to publish to avoid clobbering manual notes" + exit 1 + fi + + # For diagnostics + - name: Display structure of files to publish + run: ls -lR ./stage + + - name: Publish nightly to meshtastic.github.io + uses: peaceiris/actions-gh-pages@v4 + with: + deploy_key: ${{ secrets.DIST_PAGES_DEPLOY_KEY }} + external_repository: meshtastic/meshtastic.github.io + publish_branch: master + publish_dir: ./stage + # keep_files:false is scoped to destination_dir, so this refreshes only + # firmware-nightly/ (clearing stale nightly binaries) while sibling + # release folders stay untouched; release_notes.md is carried in above. + destination_dir: firmware-nightly + keep_files: false + user_name: github-actions[bot] + user_email: github-actions[bot]@users.noreply.github.com + commit_message: Nightly ${{ needs.version.outputs.long }} + enable_jekyll: true diff --git a/.github/workflows/models_issue_triage.yml b/.github/workflows/models_issue_triage.yml deleted file mode 100644 index a02646ea09b..00000000000 --- a/.github/workflows/models_issue_triage.yml +++ /dev/null @@ -1,213 +0,0 @@ -name: Issue Triage (Models) - -on: - issues: - types: [opened] - -permissions: - issues: write - models: read - -concurrency: - group: ${{ github.workflow }}-${{ github.event.issue.number }} - cancel-in-progress: true - -jobs: - triage: - if: ${{ github.repository == 'meshtastic/firmware' && github.event.issue.user.type != 'Bot' }} - runs-on: ubuntu-latest - steps: - # ───────────────────────────────────────────────────────────────────────── - # Step 1: Quality check (spam/AI-slop detection) - runs first, exits early if spam - # ───────────────────────────────────────────────────────────────────────── - - name: Detect spam or low-quality content - uses: actions/ai-inference@v2 - id: quality - continue-on-error: true - with: - max-tokens: 20 - prompt: | - Is this GitHub issue spam, AI-generated slop, or low quality? - - Title: ${{ github.event.issue.title }} - Body: ${{ github.event.issue.body }} - - Respond with exactly one of: spam, ai-generated, needs-review, ok - system-prompt: You detect spam and low-quality contributions. Be conservative - only flag obvious spam or AI slop. - model: openai/gpt-4o-mini - - - name: Apply quality label if needed - if: steps.quality.outputs.response != '' && steps.quality.outputs.response != 'ok' - uses: actions/github-script@v9 - env: - QUALITY_LABEL: ${{ steps.quality.outputs.response }} - with: - script: | - const label = (process.env.QUALITY_LABEL || '').trim().toLowerCase(); - const labelMeta = { - 'spam': { color: 'd73a4a', description: 'Possible spam' }, - 'ai-generated': { color: 'fbca04', description: 'Possible AI-generated low-quality content' }, - 'needs-review': { color: 'f9d0c4', description: 'Needs human review' }, - }; - const meta = labelMeta[label]; - if (!meta) return; - - // Ensure label exists - try { - await github.rest.issues.getLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label }); - } catch (e) { - if (e.status !== 404) throw e; - await github.rest.issues.createLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label, color: meta.color, description: meta.description }); - } - - // Apply label - await github.rest.issues.addLabels({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.payload.issue.number, labels: [label] }); - - // Set output to skip remaining steps - core.setOutput('is_spam', 'true'); - - # ───────────────────────────────────────────────────────────────────────── - # Step 2: Duplicate detection - only if not spam - # ───────────────────────────────────────────────────────────────────────── - - name: Detect duplicate issues - if: steps.quality.outputs.response == 'ok' || steps.quality.outputs.response == '' - uses: pelikhan/action-genai-issue-dedup@bdb3b5d9451c1090ffcdf123d7447a5e7c7a2528 # v0.0.19 - with: - github_token: ${{ secrets.GITHUB_TOKEN }} - - # ───────────────────────────────────────────────────────────────────────── - # Step 3: Completeness check + auto-labeling (combined into one AI call) - # ───────────────────────────────────────────────────────────────────────── - - name: Determine if completeness check should be skipped - if: steps.quality.outputs.response == 'ok' || steps.quality.outputs.response == '' - uses: actions/github-script@v9 - id: check-skip - with: - script: | - const title = (context.payload.issue.title || '').toLowerCase(); - const labels = (context.payload.issue.labels || []).map(label => label.name); - const hasFeatureRequest = title.includes('feature request'); - const hasEnhancement = labels.includes('enhancement'); - const shouldSkip = hasFeatureRequest && hasEnhancement; - core.setOutput('should_skip', shouldSkip ? 'true' : 'false'); - - - name: Analyze issue completeness and determine labels - if: (steps.quality.outputs.response == 'ok' || steps.quality.outputs.response == '') && steps.check-skip.outputs.should_skip != 'true' - uses: actions/ai-inference@v2 - id: analysis - continue-on-error: true - with: - prompt: | - Analyze this GitHub issue for completeness and determine if it needs labels. - - IMPORTANT: Distinguish between: - - Device/firmware bugs (crashes, reboots, lockups, radio/GPS/display/power issues) - these need device logs - - Build/release/packaging issues (missing files, CI failures, download problems) - these do NOT need device logs - - Documentation or website issues - these do NOT need device logs - - If this is a device/firmware bug, request device logs and explain how to get them: - - Web Flasher logs: - - Go to https://flasher.meshtastic.org - - Connect the device via USB and click Connect - - Open the device console/log output, reproduce the problem, then copy/download and attach/paste the logs - - Meshtastic CLI logs: - - Run: meshtastic --port --noproto - - Reproduce the problem, then copy/paste the terminal output - - Also request key context if missing: device model/variant, firmware version, region, steps to reproduce, expected vs actual. - - Respond ONLY with valid JSON (no markdown, no code fences): - {"complete": true, "comment": "", "label": "none"} - OR - {"complete": false, "comment": "Your helpful comment", "label": "needs-logs"} - - Use "needs-logs" ONLY if this is a device/firmware bug AND no logs are attached. - Use "needs-info" if basic info like firmware version or steps to reproduce are missing. - Use "none" if the issue is complete, is a feature request, or is a build/CI/packaging issue. - - Title: ${{ github.event.issue.title }} - Body: ${{ github.event.issue.body }} - system-prompt: You are a helpful assistant that triages GitHub issues. Be conservative with labels. Only request device logs for actual device/firmware bugs, not for build/release/CI issues. - model: openai/gpt-4o-mini - - - name: Process analysis result - if: (steps.quality.outputs.response == 'ok' || steps.quality.outputs.response == '') && steps.check-skip.outputs.should_skip != 'true' && steps.analysis.outputs.response != '' - uses: actions/github-script@v9 - id: process - env: - AI_RESPONSE: ${{ steps.analysis.outputs.response }} - with: - script: | - let raw = (process.env.AI_RESPONSE || '').trim(); - - // Strip markdown code fences if present - raw = raw.replace(/^```(?:json)?\s*/i, '').replace(/\s*```$/i, '').trim(); - - let complete = true; - let comment = ''; - let label = 'none'; - - try { - const parsed = JSON.parse(raw); - complete = !!parsed.complete; - comment = (parsed.comment ?? '').toString().trim(); - label = (parsed.label ?? 'none').toString().trim().toLowerCase(); - } catch { - // If JSON parse fails, log warning and don't comment (avoid posting raw JSON) - console.log('Failed to parse AI response as JSON:', raw); - complete = true; - comment = ''; - label = 'none'; - } - - // Validate label - const allowedLabels = new Set(['needs-logs', 'needs-info', 'none']); - if (!allowedLabels.has(label)) label = 'none'; - - // Only comment if we have a valid parsed comment (not raw JSON) - const shouldComment = !complete && comment.length > 0 && !comment.startsWith('{'); - core.setOutput('should_comment', shouldComment ? 'true' : 'false'); - core.setOutput('comment_body', comment); - core.setOutput('label', label); - - - name: Apply triage label - if: steps.process.outputs.label != '' && steps.process.outputs.label != 'none' - uses: actions/github-script@v9 - env: - LABEL_NAME: ${{ steps.process.outputs.label }} - with: - script: | - const label = process.env.LABEL_NAME; - const labelMeta = { - 'needs-logs': { color: 'cfd3d7', description: 'Device logs requested for triage' }, - 'needs-info': { color: 'f9d0c4', description: 'More information requested for triage' }, - }; - const meta = labelMeta[label]; - if (!meta) return; - - // Ensure label exists - try { - await github.rest.issues.getLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label }); - } catch (e) { - if (e.status !== 404) throw e; - await github.rest.issues.createLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label, color: meta.color, description: meta.description }); - } - - // Apply label - await github.rest.issues.addLabels({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.payload.issue.number, labels: [label] }); - - - name: Comment on issue - if: steps.process.outputs.should_comment == 'true' - uses: actions/github-script@v9 - env: - COMMENT_BODY: ${{ steps.process.outputs.comment_body }} - with: - script: | - await github.rest.issues.createComment({ - owner: context.repo.owner, - repo: context.repo.repo, - issue_number: context.payload.issue.number, - body: process.env.COMMENT_BODY - }); diff --git a/.github/workflows/models_pr_triage.yml b/.github/workflows/models_pr_triage.yml deleted file mode 100644 index f39ee4845fb..00000000000 --- a/.github/workflows/models_pr_triage.yml +++ /dev/null @@ -1,139 +0,0 @@ -name: PR Triage (Models) - -on: - pull_request_target: - types: [opened] - -permissions: - pull-requests: write - issues: write - models: read - -concurrency: - group: ${{ github.workflow }}-${{ github.event.pull_request.number }} - cancel-in-progress: true - -jobs: - triage: - if: ${{ github.repository == 'meshtastic/firmware' && github.event.pull_request.user.type != 'Bot' }} - runs-on: ubuntu-latest - steps: - # ───────────────────────────────────────────────────────────────────────── - # Step 1: Check if PR already has automation/type labels (skip if so) - # ───────────────────────────────────────────────────────────────────────── - - name: Check existing labels - uses: actions/github-script@v9 - id: check-labels - with: - script: | - const skipLabels = new Set(['automation']); - const typeLabels = new Set(['bugfix', 'hardware-support', 'enhancement', 'dependencies', 'submodules', 'github_actions', 'trunk', 'cleanup']); - const prLabels = context.payload.pull_request.labels.map(l => l.name); - - const shouldSkipAll = prLabels.some(l => skipLabels.has(l)); - const hasTypeLabel = prLabels.some(l => typeLabels.has(l)); - - core.setOutput('skip_all', shouldSkipAll ? 'true' : 'false'); - core.setOutput('has_type_label', hasTypeLabel ? 'true' : 'false'); - - # ───────────────────────────────────────────────────────────────────────── - # Step 2: Quality check (spam/AI-slop detection) - # ───────────────────────────────────────────────────────────────────────── - - name: Detect spam or low-quality content - if: steps.check-labels.outputs.skip_all != 'true' - uses: actions/ai-inference@v2 - id: quality - continue-on-error: true - with: - max-tokens: 20 - prompt: | - Is this GitHub pull request spam, AI-generated slop, or low quality? - - Title: ${{ github.event.pull_request.title }} - Body: ${{ github.event.pull_request.body }} - - Respond with exactly one of: spam, ai-generated, needs-review, ok - system-prompt: You detect spam and low-quality contributions. Be conservative - only flag obvious spam or AI slop. - model: openai/gpt-4o-mini - - - name: Apply quality label if needed - if: steps.check-labels.outputs.skip_all != 'true' && steps.quality.outputs.response != '' && steps.quality.outputs.response != 'ok' - uses: actions/github-script@v9 - id: quality-label - env: - QUALITY_LABEL: ${{ steps.quality.outputs.response }} - with: - script: | - const label = (process.env.QUALITY_LABEL || '').trim().toLowerCase(); - const labelMeta = { - 'spam': { color: 'd73a4a', description: 'Possible spam' }, - 'ai-generated': { color: 'fbca04', description: 'Possible AI-generated low-quality content' }, - 'needs-review': { color: 'f9d0c4', description: 'Needs human review' }, - }; - const meta = labelMeta[label]; - if (!meta) return; - - // Ensure label exists - try { - await github.rest.issues.getLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label }); - } catch (e) { - if (e.status !== 404) throw e; - await github.rest.issues.createLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label, color: meta.color, description: meta.description }); - } - - // Apply label - await github.rest.issues.addLabels({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.payload.pull_request.number, labels: [label] }); - - core.setOutput('is_spam', 'true'); - - # ───────────────────────────────────────────────────────────────────────── - # Step 3: Auto-label PR type (bugfix/hardware-support/enhancement) - # Only skip for spam/ai-generated; still classify needs-review PRs - # ───────────────────────────────────────────────────────────────────────── - - name: Classify PR for labeling - if: steps.check-labels.outputs.skip_all != 'true' && steps.check-labels.outputs.has_type_label != 'true' && steps.quality.outputs.response != 'spam' && steps.quality.outputs.response != 'ai-generated' - uses: actions/ai-inference@v2 - id: classify - continue-on-error: true - with: - max-tokens: 30 - prompt: | - Classify this pull request into exactly one category. - - Return exactly one of: bugfix, hardware-support, enhancement - - Use bugfix if it fixes a bug, crash, or incorrect behavior. - Use hardware-support if it adds or improves support for a specific hardware device/variant. - Use enhancement if it adds a new feature, improves performance, or refactors code. - - Title: ${{ github.event.pull_request.title }} - Body: ${{ github.event.pull_request.body }} - system-prompt: You classify pull requests into categories. Be conservative and pick the most appropriate single label. - model: openai/gpt-4o-mini - - - name: Apply type label - if: steps.check-labels.outputs.skip_all != 'true' && steps.check-labels.outputs.has_type_label != 'true' && steps.classify.outputs.response != '' - uses: actions/github-script@v9 - env: - TYPE_LABEL: ${{ steps.classify.outputs.response }} - with: - script: | - const label = (process.env.TYPE_LABEL || '').trim().toLowerCase(); - const labelMeta = { - 'bugfix': { color: 'd73a4a', description: 'Bug fix' }, - 'hardware-support': { color: '0e8a16', description: 'Hardware support addition or improvement' }, - 'enhancement': { color: 'a2eeef', description: 'New feature or enhancement' }, - }; - const meta = labelMeta[label]; - if (!meta) return; - - // Ensure label exists - try { - await github.rest.issues.getLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label }); - } catch (e) { - if (e.status !== 404) throw e; - await github.rest.issues.createLabel({ owner: context.repo.owner, repo: context.repo.repo, name: label, color: meta.color, description: meta.description }); - } - - // Apply label - await github.rest.issues.addLabels({ owner: context.repo.owner, repo: context.repo.repo, issue_number: context.payload.pull_request.number, labels: [label] }); diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 045e94895c7..f0a79e330e2 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -14,7 +14,7 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Trunk Check uses: trunk-io/trunk-action@v1 @@ -31,7 +31,7 @@ jobs: pull-requests: write # For trunk to create PRs steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Trunk Upgrade uses: trunk-io/trunk-action/upgrade@v1 diff --git a/.github/workflows/package_obs.yml b/.github/workflows/package_obs.yml index b491f006251..520c0851ecf 100644 --- a/.github/workflows/package_obs.yml +++ b/.github/workflows/package_obs.yml @@ -33,7 +33,7 @@ jobs: needs: build-debian-src steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive path: meshtasticd diff --git a/.github/workflows/package_pio_deps.yml b/.github/workflows/package_pio_deps.yml index 6bd256f52cc..bf2576a53e4 100644 --- a/.github/workflows/package_pio_deps.yml +++ b/.github/workflows/package_pio_deps.yml @@ -23,7 +23,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/package_ppa.yml b/.github/workflows/package_ppa.yml index c51e64e783e..88d98e628af 100644 --- a/.github/workflows/package_ppa.yml +++ b/.github/workflows/package_ppa.yml @@ -34,7 +34,7 @@ jobs: needs: build-debian-src steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: recursive path: meshtasticd diff --git a/.github/workflows/pr_tests.yml b/.github/workflows/pr_tests.yml index e3321712e8a..7d910154cf9 100644 --- a/.github/workflows/pr_tests.yml +++ b/.github/workflows/pr_tests.yml @@ -40,7 +40,7 @@ jobs: checks: write pull-requests: write steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: submodules: recursive diff --git a/.github/workflows/release_channels.yml b/.github/workflows/release_channels.yml index a85979c725a..3184413ef37 100644 --- a/.github/workflows/release_channels.yml +++ b/.github/workflows/release_channels.yml @@ -91,7 +91,7 @@ jobs: shell: bash steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: # Always use master branch for version bumps ref: master diff --git a/.github/workflows/sec_sast_semgrep_cron.yml b/.github/workflows/sec_sast_semgrep_cron.yml index 95e5c2c3d86..da5d60a893a 100644 --- a/.github/workflows/sec_sast_semgrep_cron.yml +++ b/.github/workflows/sec_sast_semgrep_cron.yml @@ -21,7 +21,7 @@ jobs: steps: # step 1 - name: clone application source code - uses: actions/checkout@v6 + uses: actions/checkout@v7 # step 2 - name: full scan diff --git a/.github/workflows/sec_sast_semgrep_pull.yml b/.github/workflows/sec_sast_semgrep_pull.yml index e9b4108a19d..1508e082256 100644 --- a/.github/workflows/sec_sast_semgrep_pull.yml +++ b/.github/workflows/sec_sast_semgrep_pull.yml @@ -13,7 +13,7 @@ jobs: steps: # step 1 - name: clone application source code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: fetch-depth: 0 diff --git a/.github/workflows/stale_bot.yml b/.github/workflows/stale_bot.yml index 9255975a8fc..576da06baa5 100644 --- a/.github/workflows/stale_bot.yml +++ b/.github/workflows/stale_bot.yml @@ -17,7 +17,7 @@ jobs: steps: - name: Stale PR+Issues - uses: actions/stale@v10.2.0 + uses: actions/stale@v10.4.0 with: days-before-stale: 45 stale-issue-message: This issue has not had any comment or update in the last month. If it is still relevant, please post update comments. If no comments are made, this issue will be closed automagically in 7 days. diff --git a/.github/workflows/test_native.yml b/.github/workflows/test_native.yml index 2fabf0591ed..3b1b95ee8a0 100644 --- a/.github/workflows/test_native.yml +++ b/.github/workflows/test_native.yml @@ -14,7 +14,7 @@ jobs: name: Native Simulator Tests runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: submodules: recursive @@ -37,13 +37,33 @@ jobs: sed -i -e "s#${PWD}#.#" coverage_base.info # Make paths relative. - name: Integration test + # Cap the whole step: if the simulator ever fails to exit (e.g. the + # exit_simulator admin path regresses again) the job must fail fast, + # not run to GitHub's 6-hour limit. + timeout-minutes: 5 run: | .pio/build/coverage/meshtasticd -s & PID=$! + trap 'kill "$PID" 2>/dev/null || true' EXIT timeout 20 bash -c "until ls -al /proc/$PID/fd | grep socket; do sleep 1; done" echo "Simulator started, launching python test..." python3 -c 'from meshtastic.test import testSimulator; testSimulator()' - wait + # The Python harness sends exit_simulator and exits; the simulator is + # expected to terminate on its own. Give it a moment, then verify. + # If it is still alive the exit handshake is broken - fail loudly and + # do NOT fall through to `wait`, which would otherwise block until the + # job's hard timeout. + for i in $(seq 1 10); do + kill -0 "$PID" 2>/dev/null || break + sleep 1 + done + if kill -0 "$PID" 2>/dev/null; then + echo "::error title=Simulator did not exit::meshtasticd ignored exit_simulator and is still running after the integration test. The exit_simulator admin path is broken (see AdminModule::handleReceivedProtobuf, ARCH_PORTDUINO bypass). Killing it to avoid a 6-hour CI overrun." + kill -9 "$PID" 2>/dev/null || true + wait "$PID" 2>/dev/null || true + exit 1 + fi + wait "$PID" 2>/dev/null || true - name: Capture coverage information if: always() # run this step even if previous step failed @@ -68,7 +88,7 @@ jobs: name: Native PlatformIO Tests runs-on: ubuntu-latest steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 with: submodules: recursive @@ -86,7 +106,13 @@ jobs: run: sed -i 's/-DBUILD_EPOCH=$UNIX_TIME/#-DBUILD_EPOCH=$UNIX_TIME/' platformio.ini - name: PlatformIO Tests - run: platformio test -e coverage -v --junit-output-path testreport.xml + run: | + set -o pipefail + # Filter out SKIPPED summary rows for hardware variants that can't run on the + # native host. They flood the log and make it harder to spot real failures. + # The JUnit XML is written directly to testreport.xml before the pipe, so + # the test artifact is unaffected. + platformio test -e coverage -v --junit-output-path testreport.xml 2>&1 | grep -v "[[:space:]]SKIPPED$" - name: Save test results if: always() # run this step even if previous step failed @@ -123,7 +149,7 @@ jobs: - platformio-tests if: always() steps: - - uses: actions/checkout@v6 + - uses: actions/checkout@v7 - name: Get release version string run: echo "long=$(./bin/buildinfo.py long)" >> $GITHUB_OUTPUT @@ -135,6 +161,14 @@ jobs: name: platformio-test-report-${{ steps.version.outputs.long }} merge-multiple: true + - name: Drop no-status testsuites from the report + # PlatformIO emits a self-closing row for every test_* dir + # crossed with every hardware variant it cannot run on the native host (~4900 rows). + # They carry no pass/fail/skip status and bury the suites that actually ran. Strip + # them so the Test Report lists only suites with a real status. Only the copy the + # reporter renders is trimmed; the uploaded artifact keeps the full XML. + run: sed -i -E 's#]*tests="0"[^>]*/>##g' testreport.xml + - name: Test Report uses: dorny/test-reporter@v3.0.0 with: diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index be51428437b..777193a466c 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -20,7 +20,7 @@ jobs: runs-on: test-runner steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 # - uses: actions/setup-python@v6 # with: @@ -47,7 +47,7 @@ jobs: pio upgrade - name: Setup Node - uses: actions/setup-node@v6 + uses: actions/setup-node@v7 with: node-version: 24 diff --git a/.github/workflows/trunk_annotate_pr.yml b/.github/workflows/trunk_annotate_pr.yml index 59ab25c2810..09a606cc890 100644 --- a/.github/workflows/trunk_annotate_pr.yml +++ b/.github/workflows/trunk_annotate_pr.yml @@ -18,7 +18,7 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Trunk Check uses: trunk-io/trunk-action@v1 diff --git a/.github/workflows/trunk_check.yml b/.github/workflows/trunk_check.yml index 874374fe0dc..acdc8ee1e9a 100644 --- a/.github/workflows/trunk_check.yml +++ b/.github/workflows/trunk_check.yml @@ -16,7 +16,7 @@ jobs: steps: - name: Checkout - uses: actions/checkout@v6 + uses: actions/checkout@v7 - name: Trunk Check uses: trunk-io/trunk-action@v1 diff --git a/.github/workflows/update_protobufs.yml b/.github/workflows/update_protobufs.yml index e9380467eac..657d9171c1c 100644 --- a/.github/workflows/update_protobufs.yml +++ b/.github/workflows/update_protobufs.yml @@ -11,18 +11,23 @@ jobs: pull-requests: write steps: - name: Checkout code - uses: actions/checkout@v6 + uses: actions/checkout@v7 with: submodules: true - name: Update submodule - if: ${{ github.ref == 'refs/heads/master' || github.ref == 'refs/heads/develop' }} + if: ${{ github.ref_name == 'master' || github.ref_name == 'develop' }} + working-directory: protobufs + env: + # Use the branch that triggered the workflow as the protobuf branch. + GIT_BRANCH: ${{ github.ref_name }} run: | - git submodule update --remote protobufs + git fetch --prune origin $GIT_BRANCH + git checkout FETCH_HEAD - name: Download nanopb run: | - wget https://jpa.kapsi.fi/nanopb/download/nanopb-0.4.9.1-linux-x86.tar.gz + wget https://github.com/nanopb/nanopb/releases/download/nanopb-0.4.9.1/nanopb-0.4.9.1-linux-x86.tar.gz tar xvzf nanopb-0.4.9.1-linux-x86.tar.gz mv nanopb-0.4.9.1-linux-x86 nanopb-0.4.9 @@ -33,7 +38,7 @@ jobs: - name: Create pull request uses: peter-evans/create-pull-request@v8 with: - branch: create-pull-request/update-protobufs + branch: create-pull-request/update-protobufs-${{ github.ref_name }} labels: submodules title: Update protobufs and classes commit-message: Update protobufs diff --git a/.gitignore b/.gitignore index f1eb9d852d7..f29da0ac6e5 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ .pio pio +.pio-docker pio.tar web web.tar @@ -47,6 +48,10 @@ data/boot/logo.* managed_components/* arduino-lib-builder* dependencies.lock + +# JLink / RTT debug artifacts (nRF SoCs) +flash.jlink +rtt_*.txt idf_component.yml CMakeLists.txt /sdkconfig.* @@ -56,3 +61,17 @@ CMakeLists.txt .python3 .claude/scheduled_tasks.lock userPrefs.jsonc.mcp-session-bak + +# Fake-NodeDB fixture pipeline (bin/regen-fake-nodedbs.sh) +# JSONL seeds are committed (test/fixtures/nodedb/seed_v25_*.jsonl); +# compiled .proto outputs are ephemeral build artifacts. +build/fixtures/ +bin/_generated/ + +# portduino WASM build (pio run -e native-wasm) + Emscripten SDK. The artifacts land in +# .pio/build/native-wasm/ (already ignored via .pio/); these cover the retired +# standalone emcc build's output dirs and any in-repo emsdk checkout. +build/wasm/ +build/wasm-obj/ +build/wasm-*.log +.emsdk/ diff --git a/.mcp.json b/.mcp.json index c5cf2e55e5a..82216666480 100644 --- a/.mcp.json +++ b/.mcp.json @@ -1,8 +1,12 @@ { "mcpServers": { "meshtastic": { - "command": "./mcp-server/.venv/bin/python", - "args": ["-m", "meshtastic_mcp"], + "command": "uvx", + "args": [ + "--from", + "git+https://github.com/meshtastic/meshtastic-mcp", + "meshtastic-mcp" + ], "env": { "MESHTASTIC_FIRMWARE_ROOT": "." } diff --git a/.semgrepignore b/.semgrepignore index b4267ad2364..fff35f6b7b0 100644 --- a/.semgrepignore +++ b/.semgrepignore @@ -1,2 +1,6 @@ .github/workflows/main_matrix.yml src/mesh/compression/unishox2.cpp +# Emscripten/WebUSB browser glue for the wasm node — not part of the firmware +# binary or its security surface. The format-string rule false-positives on its +# benign retry/diagnostic console logs. +src/platform/portduino/wasm/js/ diff --git a/.trunk/configs/.bandit b/.trunk/configs/.bandit index c70e7743b67..cca791bb458 100644 --- a/.trunk/configs/.bandit +++ b/.trunk/configs/.bandit @@ -12,10 +12,10 @@ # defensive loops over flaky sources (pubsub handlers, device # re-enumeration polls). One failed iteration shouldn't abort the loop. # B404 import_subprocess -# mcp-server wraps PlatformIO, esptool, nrfutil, picotool, and the -# pytest test-runner — subprocess is a load-bearing import here, not -# a smell. The "consider possible security implications" advisory is -# redundant given the file-level review already applied. +# the bin/ proto + fixture tooling (regen-py-protos, gen-fake-nodedb-seed, +# seed-json-to-proto) shells out to protoc and helper scripts — subprocess +# is a load-bearing import here, not a smell. The "consider possible security +# implications" advisory is redundant given the file-level review already applied. # B603 subprocess_without_shell_equals_true # all subprocess calls use a static argv list; `shell=False` is the # default and we never string-interpolate user input into the command. diff --git a/.trunk/trunk.yaml b/.trunk/trunk.yaml index 0bbfbcf08e6..9b0c760bbfd 100644 --- a/.trunk/trunk.yaml +++ b/.trunk/trunk.yaml @@ -7,7 +7,68 @@ plugins: ref: v1.10.0 uri: https://github.com/trunk-io/plugins lint: + # Custom file set + formatter that rewrites Unicode em-dash (U+2014, UTF-8 + # bytes e2 80 94) and en-dash (U+2013, UTF-8 bytes e2 80 93) to an ASCII + # hyphen. The sed pattern matches the byte sequences under LC_ALL=C rather + # than literal characters on purpose, so the rule survives editors/hooks + # that themselves strip Unicode dashes. These dashes only ever appear in + # comments and log/error strings (LLM-authored prose), never in code, so + # the rewrite is purely cosmetic. Runs under `trunk fmt`; enforced by + # `trunk check` and the trunk-fmt-pre-commit action. + files: + - name: ascii-dash-targets + extensions: + - c + - cc + - cpp + - h + - hpp + - ino + - md + - py + - sh + - yaml + - yml + - json + - toml + - ini + - name: cpp-sources + extensions: + - c + - cc + - cpp + - cxx + - h + - hh + - hpp + - hxx + - ino + definitions: + - name: ascii-dash + files: [ascii-dash-targets] + commands: + - name: format + output: rewrite + formatter: true + run: env LC_ALL=C sed -e 's/\xe2\x80\x94/-/g' -e 's/\xe2\x80\x93/-/g' ${target} + success_codes: [0] + # Flags preprocessor conditionals that contain too many defined() terms + # (e.g. the 13-driver display chains in main.cpp). The compiler has no + # warning for this; the fix is an umbrella feature macro (HAS_TFT / + # HAS_SCREEN, already in src/configuration.h). Tune the limit via + # MAX_DEFINED in bin/lint-ifdef-complexity.sh (default 5 -> warns at 6+). + - name: too-many-defined + files: [cpp-sources] + commands: + - name: lint + output: regex + parse_regex: (?P.+):(?P\d+):(?P\d+):(?P\w+):(?P.+):(?P[a-z-]+) + run: ${workspace}/bin/lint-ifdef-complexity.sh ${target} + success_codes: [0] + read_output_from: stdout enabled: + - ascii-dash@SYSTEM + - too-many-defined@SYSTEM - checkov@3.2.529 - renovate@43.150.0 - prettier@3.8.3 @@ -34,6 +95,19 @@ lint: - linters: [ALL] paths: - bin/** + # Fake-NodeDB fixture JSONL files contain deterministic synthetic + # public_key_hex (64-char hex) values that gitleaks misidentifies as + # generic-api-key. These are not secrets - they're test fixtures + # produced by bin/gen-fake-nodedb-seed.py with a fixed RNG seed. + - linters: [gitleaks] + paths: + - test/fixtures/nodedb/seed_v25_*.jsonl + # The UA font's em/en dashes live only in trailing comments documenting + # each glyph's codepoint (e.g. "8212=:2549"); rewriting them would make + # those docs inaccurate. Glyph byte data is unaffected either way. + - linters: [ascii-dash] + paths: + - src/graphics/fonts/** runtimes: enabled: - python@3.14.4 diff --git a/AGENTS.md b/AGENTS.md index 82912f252f4..9423c516927 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,73 +1,87 @@ # Agent instructions -This repository is the [Meshtastic](https://meshtastic.org) firmware — a C++17 embedded codebase targeting ESP32 / nRF52 / RP2040 / STM32WL / Linux-Portduino LoRa mesh radios — plus a Python MCP server in `mcp-server/` that AI agents use to flash, configure, and test connected devices. +> **TL;DR** +> +> | | | +> | -------------- | ---------------------------------------------------------------------------------------------------------------------- | +> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) | +> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | +> | Format | `trunk fmt` | +> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `CLAUDE.md` (Claude Code) | +> +> **Need this? It's here.** +> +> | | | +> | ------------------------------------------- | ---------------------------------------------------------- | +> | General helpers (clamp, UTF-8, string fmt…) | `src/meshUtils.h` | +> | Logging macros (LOG_DEBUG / INFO / WARN…) | `src/DebugConfiguration.h` | +> | New module skeleton | inherit `ProtobufModule` in `src/mesh/ProtobufModule.h` | +> | Observer / event wiring | `src/Observer.h` | + +This repository is the [Meshtastic](https://meshtastic.org) firmware - a C++17 embedded codebase targeting ESP32 / nRF52 / RP2040 / STM32WL / Linux-Portduino LoRa mesh radios. The Python MCP server that AI agents use to flash, configure, and test connected devices now lives in its own repo, [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp); this repo registers it via `.mcp.json` (run through `uvx`) so its tools are available automatically. ## Primary instruction file -**Read `.github/copilot-instructions.md` first.** That file is the canonical agent-facing document for this repo. It covers project layout, coding conventions (naming, module framework, Observer pattern, thread safety), the build system, CI/CD, the native C++ test suite, and — most importantly for automation work — the **MCP Server & Hardware Test Harness** section. Read it top-to-bottom before starting any non-trivial change. +**Read `.github/copilot-instructions.md` first.** That file is the canonical agent-facing document for this repo. It covers project layout, coding conventions (naming, module framework, Observer pattern, thread safety), the build system, CI/CD, the native C++ test suite, and - most importantly for automation work - the **MCP Server & Hardware Test Harness** section. Read it top-to-bottom before starting any non-trivial change. This file (`AGENTS.md`) is a short pointer + quick reference for agents that don't read `.github/copilot-instructions.md` by default. ## Quick command reference -| Action | Command | -| -------------------------------- | ------------------------------------------------------------------------------------------------------------- | -| Build a firmware variant | `pio run -e ` (e.g. `pio run -e rak4631`, `pio run -e heltec-v3`) | -| Build native macOS host binary | `pio run -e native-macos` (Homebrew prereqs + CH341 LoRa setup in `variants/native/portduino/platformio.ini`) | -| Clean + rebuild | `pio run -e -t clean && pio run -e ` | -| Flash a device | `pio run -e -t upload --upload-port ` (or use the `pio_flash` MCP tool) | -| Run firmware unit tests (native) | `pio test -e native` | -| Run MCP hardware tests | `./mcp-server/run-tests.sh` | -| Live TUI test runner | `mcp-server/.venv/bin/meshtastic-mcp-test-tui` | -| Format before commit | `trunk fmt` | -| Regenerate protobuf bindings | `bin/regen-protos.sh` | -| Generate CI matrix | `./bin/generate_ci_matrix.py all [--level pr]` | +| Action | Command | +| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Build a firmware variant | `pio run -e ` (e.g. `pio run -e rak4631`, `pio run -e heltec-v3`) | +| Build native macOS host binary | `pio run -e native-macos` (Homebrew prereqs + CH341 LoRa setup in `variants/native/portduino/platformio.ini`) | +| Clean + rebuild | `pio run -e -t clean && pio run -e ` | +| Flash a device | `pio run -e -t upload --upload-port ` (or use the `pio_flash` MCP tool) | +| Run firmware unit tests (native) | `./bin/run-tests.sh` (preferred - ASan/LSan + RED/AMBER/GREEN verdict); or raw: `~/.platformio/penv/bin/python -m platformio test -e native > /tmp/test_out.txt 2>&1` | +| Run MCP hardware tests | From a [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) checkout: `MESHTASTIC_FIRMWARE_ROOT=/path/to/firmware ./run-tests.sh` | +| Live TUI test runner | `uvx --from git+https://github.com/meshtastic/meshtastic-mcp meshtastic-mcp-test-tui` | +| Format before commit | `trunk fmt` | +| Regenerate protobuf bindings | `bin/regen-protos.sh` | +| Generate CI matrix | `./bin/generate_ci_matrix.py all [--level pr]` | ## MCP server (device + test automation) -The `mcp-server/` package exposes ~32 MCP tools for device discovery, building, flashing, serial monitoring, and live-node administration. Tools are grouped as: +The [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) server exposes ~32 MCP tools for device discovery, building, flashing, serial monitoring, and live-node administration. Tools are grouped as: - **Discovery**: `list_devices`, `list_boards`, `get_board` - **Build & flash**: `build`, `clean`, `pio_flash`, `erase_and_flash` (ESP32 factory), `update_flash` (ESP32 OTA), `touch_1200bps` - **Serial sessions**: `serial_open`, `serial_read`, `serial_list`, `serial_close` - **Device reads**: `device_info`, `list_nodes` - **Device writes** (require `confirm=True`): `set_owner`, `get_config`, `set_config`, `get_channel_url`, `set_channel_url`, `send_text`, `reboot`, `shutdown`, `factory_reset`, `set_debug_log_api` +- **Frame injection**: `inject_frame` - deliver a crafted frame into a board's real RX pipeline as if received off LoRa (reaches `from != 0` / decrypt / remote-admin paths the `toRadio` API can't). Needs firmware built with `-D MESHTASTIC_ENABLE_FRAME_INJECTION=1` (`MeshService::injectAsReceived`); sim nodes always. See the copilot-instructions **Frame injection** section. - **userPrefs admin**: `userprefs_get`, `userprefs_set`, `userprefs_reset`, `userprefs_manifest`, `userprefs_testing_profile` - **Vendor escape hatches**: `esptool_*`, `nrfutil_*`, `picotool_*` -Setup: `cd mcp-server && python3 -m venv .venv && .venv/bin/pip install -e '.[test]'`. The repo registers the server via `.mcp.json` — Claude Code picks it up automatically. +Setup: nothing to build - `.mcp.json` runs the server via `uvx --from git+https://github.com/meshtastic/meshtastic-mcp meshtastic-mcp`, so Claude Code picks it up automatically. To run the pytest hardware harness, clone [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) and set `MESHTASTIC_FIRMWARE_ROOT` to this firmware checkout. -See `mcp-server/README.md` for argument shapes and the **MCP Server & Hardware Test Harness** section of `.github/copilot-instructions.md` for agent usage rules (tool surface, fixture contract, firmware integration points, recovery playbooks). +See the meshtastic-mcp repo's README for argument shapes and the **MCP Server & Hardware Test Harness** section of `.github/copilot-instructions.md` for agent usage rules (tool surface, fixture contract, firmware integration points, recovery playbooks). ## Slash commands (AI-assisted workflows) -Three test-and-diagnose workflows exist as slash commands: - -- **`/test` (Claude Code) / `/mcp-test` (Copilot)** — run the hardware test suite and interpret failures -- **`/diagnose` / `/mcp-diagnose`** — read-only device health report -- **`/repro` / `/mcp-repro`** — flakiness triage: re-run one test N times, diff firmware logs between passes and failures - -Bodies live in `.claude/commands/` and `.github/prompts/` respectively. `.claude/commands/README.md` is the index. +The test-and-diagnose workflows (`/test`, `/diagnose`, `/repro`, `/leakhunt`) now ship with the [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) repo as skills - run them from a checkout of that repo (with `MESHTASTIC_FIRMWARE_ROOT` pointed here). The MCP tools they build on are still available in this repo via `.mcp.json`. ## Encryption at a glance Two layers, both in `src/mesh/CryptoEngine.cpp`: -- **Channel (symmetric)** — **AES-CTR** with a channel-wide PSK (AES-128 or AES-256). Nonce = packet_id ‖ from_node ‖ block_counter. No AEAD; integrity is soft (channel-hash filter). The well-known default PSK lives in `src/mesh/Channels.h`; a 1-byte PSK is a short-form index into it. -- **Per-peer PKI** — **X25519 ECDH** (Curve25519, 32-byte keys) → SHA-256 → **AES-256-CCM** with an 8-byte MAC. Fresh 32-bit `extraNonce` per packet, sent in the clear alongside the MAC. 12-byte wire overhead (`MESHTASTIC_PKC_OVERHEAD`). Used for DMs. Also used for remote admin (`src/modules/AdminModule.cpp`), where AdminMessage authorization is gated by `config.security.admin_key[0..2]`. Disabled entirely in Ham mode (`user.is_licensed=true`). +- **Channel (symmetric)** - **AES-CTR** with a channel-wide PSK (AES-128 or AES-256). Nonce = packet_id ‖ from_node ‖ block_counter. No AEAD; integrity is soft (channel-hash filter). The well-known default PSK lives in `src/mesh/Channels.h`; a 1-byte PSK is a short-form index into it. +- **Per-peer PKI** - **X25519 ECDH** (Curve25519, 32-byte keys) → SHA-256 → **AES-256-CCM** with an 8-byte MAC. Fresh 32-bit `extraNonce` per packet, sent in the clear alongside the MAC. 12-byte wire overhead (`MESHTASTIC_PKC_OVERHEAD`). Used for DMs. Also used for remote admin (`src/modules/AdminModule.cpp`), where AdminMessage authorization is gated by `config.security.admin_key[0..2]`. Disabled entirely in Ham mode (`user.is_licensed=true`). -Key rotation to never trigger casually: only the **full** factory reset (`factory_reset_device`, `eraseBleBonds=true`) wipes `security.private_key` and regenerates the keypair — every peer holds the old public key, so DMs silently fail PKI decrypt until NodeInfo re-exchanges. The **partial** config reset (`factory_reset_config`) preserves the private key and doesn't invalidate peer relationships. Explicitly blanking `security.private_key` via admin also triggers regen. See the **Encryption & Key Management** section of `.github/copilot-instructions.md` for the full spec (nonce layout, send/receive selection logic including infrastructure-portnum exceptions, admin-key + session-passkey authorization, `is_managed` scope, key-rotation hazards). +Key rotation to never trigger casually: only the **full** factory reset (`factory_reset_device`, `eraseBleBonds=true`) wipes `security.private_key` and regenerates the keypair - every peer holds the old public key, so DMs silently fail PKI decrypt until NodeInfo re-exchanges. The **partial** config reset (`factory_reset_config`) preserves the private key and doesn't invalidate peer relationships. Explicitly blanking `security.private_key` via admin also triggers regen. See the **Encryption & Key Management** section of `.github/copilot-instructions.md` for the full spec (nonce layout, send/receive selection logic including infrastructure-portnum exceptions, admin-key + session-passkey authorization, `is_managed` scope, key-rotation hazards). ## House rules -- **No destructive device operations without operator approval.** `factory_reset`, `erase_and_flash`, `reboot`, `shutdown`, history-rewriting git ops — describe the action and stop. Operator authorizes. +- **No destructive device operations without operator approval.** `factory_reset`, `erase_and_flash`, `reboot`, `shutdown`, history-rewriting git ops - describe the action and stop. Operator authorizes. - **One MCP call per serial port at a time.** The port lock is exclusive; concurrent calls deadlock. Sequence: open → read/mutate → close, then next device. - **`userPrefs.jsonc` is session state during tests.** The `_session_userprefs` fixture snapshots + restores it; never edit it from inside a test. - **Don't speculate about firmware root causes.** When evidence doesn't support a classification, say "unknown" and list what would disambiguate. -- **Run `trunk fmt` before proposing a commit.** The `trunk_check` CI gate will reject unformatted code. +- **Run `trunk fmt` before proposing a commit.** The `trunk_check` CI gate will reject unformatted code. Claude Code runs it automatically via the PostToolUse hook in `.claude/settings.json`; trunk's launcher needs `curl` or `wget` to bootstrap its pinned CLI - see **Formatting & the trunk toolchain** in `.github/copilot-instructions.md` for the no-curl bootstrap procedure. +- **Never edit or commit files under `src/mesh/generated/`.** They are regenerated from the [`meshtastic/protobufs`](https://github.com/meshtastic/protobufs) repo by the `update_protobufs.yml` workflow (entry point: `bin/regen-protos.sh`). Local edits will be overwritten and create merge conflicts. If a `.proto` change is needed, open a PR against the protobufs repo first, then let the workflow re-sync this repo. - **`confirm=True` on destructive MCP tools is a real gate, not a formality.** Don't bypass it via auto-approve settings. -- **Keep code comments minimal — one or two lines, max.** Comment only when the _why_ isn't obvious from the code; never restate what the next line does. No multi-paragraph block comments explaining straightforward changes. The diff and commit message carry the rationale; the code carries the behavior. -- **Use `Throttle` for time-based rate limiting, not raw `millis()` math.** `src/mesh/Throttle.h` provides `Throttle::isWithinTimespanMs(lastMs, intervalMs)` (returns true while inside the cooldown) and `Throttle::execute(&lastMs, intervalMs, func)` (function-pointer form that updates the timestamp on fire). Use these for any "did N ms pass since X" check — raw `millis() > lastMs + N` is rollover-unsafe (breaks after ~49.7 days) and inconsistent with the rest of the codebase. The helpers compute `now - lastMs` with unsigned subtraction, which wraps correctly. +- **Keep code comments minimal - one or two lines, max.** Comment only when the _why_ isn't obvious from the code; never restate what the next line does. No multi-paragraph block comments explaining straightforward changes. The diff and commit message carry the rationale; the code carries the behavior. +- **Use `Throttle` for time-based rate limiting, not raw `millis()` math.** `src/mesh/Throttle.h` provides `Throttle::isWithinTimespanMs(lastMs, intervalMs)` (returns true while inside the cooldown) and `Throttle::execute(&lastMs, intervalMs, func)` (function-pointer form that updates the timestamp on fire). Use these for any "did N ms pass since X" check - raw `millis() > lastMs + N` is rollover-unsafe (breaks after ~49.7 days) and inconsistent with the rest of the codebase. The helpers compute `now - lastMs` with unsigned subtraction, which wraps correctly. ## Typical agent workflows @@ -79,9 +93,9 @@ Key rotation to never trigger casually: only the **full** factory reset (`factor ### Inspecting live node state -1. `device_info(port=...)` — short summary (node num, firmware version, region, peer count) -2. `list_nodes(port=...)` — full peer table (SNR, RSSI, pubkey presence, last_heard) -3. `get_config(section="lora", port=...)` — LoRa settings for cross-device comparison +1. `device_info(port=...)` - short summary (node num, firmware version, region, peer count) +2. `list_nodes(port=...)` - full peer table (SNR, RSSI, pubkey presence, last_heard) +3. `get_config(section="lora", port=...)` - LoRa settings for cross-device comparison Sequence these; don't parallelize on the same port. @@ -89,42 +103,40 @@ Sequence these; don't parallelize on the same port. 1. Build locally: `pio run -e ` 2. Flash the test device: `pio_flash(env=..., port=..., confirm=True)` -3. Run the suite: `./mcp-server/run-tests.sh tests/` or `/test tests/` -4. On failure, open `mcp-server/tests/report.html` → `Meshtastic debug` section for the firmware log tail + device state dump +3. Run the suite from a meshtastic-mcp checkout: `MESHTASTIC_FIRMWARE_ROOT=/path/to/firmware ./run-tests.sh tests/` (or the `/test` skill) +4. On failure, open the run's `tests/report.html` → `Meshtastic debug` section for the firmware log tail + device state dump 5. Iterate ### Debugging a flaky test -1. `/repro [count]` — re-runs the test N times, diffs firmware logs between passes and failures +1. `/repro [count]` - re-runs the test N times, diffs firmware logs between passes and failures 2. If the first attempt always fails and the rest pass, that's a state-leak pattern → suggest `--force-bake` or a clean device state, don't chase the first failure -3. If all N fail, this isn't a flake — it's a regression. Stop iterating and escalate to `/test` for full-suite context. +3. If all N fail, this isn't a flake - it's a regression. Stop iterating and escalate to `/test` for full-suite context. ## Where to look -| Path | What's there | -| --------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | -| `src/` | Firmware C++ source (`mesh/`, `modules/`, `platform/`, `graphics/`, `gps/`, `motion/`, `mqtt/`, …) | -| `src/mesh/` | Core: NodeDB, Router, Channels, CryptoEngine, radio interfaces, StreamAPI, PhoneAPI | -| `src/modules/` | Feature modules; `Telemetry/Sensor/` has 50+ I2C sensor drivers | -| `variants/` | 200+ hardware variant definitions (`variant.h` + `platformio.ini` per board) | -| `protobufs/` | `.proto` definitions; regenerate with `bin/regen-protos.sh` | -| `test/` | Firmware unit tests (12 suites; `pio test -e native`) | -| `mcp-server/` | Python MCP server + pytest hardware integration tests | -| `mcp-server/tests/` | Tiered pytest suite: `unit/`, `mesh/`, `telemetry/`, `monitor/`, `recovery/`, `ui/`, `fleet/`, `admin/`, `provisioning/` | -| `.claude/commands/` | Claude Code slash command bodies | -| `.github/prompts/` | Copilot prompt bodies (mirrors of the Claude Code ones) | -| `.github/copilot-instructions.md` | **Primary agent instructions — read this** | -| `.github/workflows/` | CI pipelines | -| `.mcp.json` | MCP server registration for Claude Code | +| Path | What's there | +| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `src/` | Firmware C++ source (`mesh/`, `modules/`, `platform/`, `graphics/`, `gps/`, `motion/`, `mqtt/`, …) | +| `src/mesh/` | Core: NodeDB, Router, Channels, CryptoEngine, radio interfaces, StreamAPI, PhoneAPI | +| `src/modules/` | Feature modules; `Telemetry/Sensor/` has 50+ I2C sensor drivers | +| `variants/` | 200+ hardware variant definitions (`variant.h` + `platformio.ini` per board) | +| `protobufs/` | `.proto` definitions; regenerate with `bin/regen-protos.sh` | +| `test/` | Firmware unit tests (19 suites; `./bin/run-tests.sh` preferred, falls back to `pio test -e native`) | +| [meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) | Standalone MCP server + tiered pytest hardware harness (`unit/`, `mesh/`, `telemetry/`, `monitor/`, `recovery/`, `ui/`, `fleet/`, `admin/`, `provisioning/`) - registered here via `.mcp.json` | +| `.github/prompts/` | Copilot prompt bodies (firmware scaffolding: new module / sensor / variant) | +| `.github/copilot-instructions.md` | **Primary agent instructions - read this** | +| `.github/workflows/` | CI pipelines | +| `.mcp.json` | MCP server registration for Claude Code | ## Recovery one-liners -- **`userPrefs.jsonc` dirty after a test run?** Re-run `./mcp-server/run-tests.sh` once (pre-flight self-heals from the sidecar). If still dirty: `git checkout userPrefs.jsonc`. +- **`userPrefs.jsonc` dirty after a test run?** Re-run the meshtastic-mcp harness once (pre-flight self-heals from the sidecar). If still dirty: `git checkout userPrefs.jsonc`. - **nRF52 not responding?** `mcp__meshtastic__touch_1200bps(port=...)` drops it into the DFU bootloader, then `pio_flash` re-installs. - **Device fully wedged (no DFU)?** `mcp__meshtastic__uhubctl_cycle(role="nrf52", confirm=True)` hard-power-cycles it via USB hub PPPS. Needs `uhubctl` installed (`brew install uhubctl` / `apt install uhubctl`); on Linux without udev rules, permission errors fail fast, so use `sudo uhubctl` yourself or configure udev access. - **Port busy?** `lsof ` to find the holder. Usually a stale `pio device monitor` or zombie `meshtastic_mcp` process. Kill it. -- **Multiple MCP servers running?** `ps aux | grep meshtastic_mcp` — zombies hold ports. Kill all but the one your host spawned. -- **macOS: `LIBUSB_ERROR_BUSY` on a CH341 LoRa adapter?** A third-party WCH `CH34xVCPDriver` is claiming interface 0. Find the bundle ID with `ioreg -p IOUSB -l -w 0 | grep -B2 -A30 0x5512`, then `sudo kmutil unload -b `. Apple's bundled CH34x kext targets the CH340 UART (PID 0x7523), not the SPI bridge — it's never the culprit. +- **Multiple MCP servers running?** `ps aux | grep meshtastic_mcp` - zombies hold ports. Kill all but the one your host spawned. +- **macOS: `LIBUSB_ERROR_BUSY` on a CH341 LoRa adapter?** A third-party WCH `CH34xVCPDriver` is claiming interface 0. Find the bundle ID with `ioreg -p IOUSB -l -w 0 | grep -B2 -A30 0x5512`, then `sudo kmutil unload -b `. Apple's bundled CH34x kext targets the CH340 UART (PID 0x7523), not the SPI bridge - it's never the culprit. ## Environment variables (test harness) @@ -135,7 +147,7 @@ Sequence these; don't parallelize on the same port. | `MESHTASTIC_MCP_FLASH_LOG` | File path to tee pio/esptool/nrfutil/picotool output. `run-tests.sh` sets this to `tests/flash.log` so the TUI can stream live flash progress. | | `MESHTASTIC_MCP_TCP_HOST` | `host` or `host:port` of a `meshtasticd` daemon (e.g. the `native-macos` build). Surfaces it in `list_devices` as `tcp://host:port` so `connect()`-based tools target it transparently. Default port 4403. | | `MESHTASTIC_UHUBCTL_BIN` | Absolute path to `uhubctl` binary. Default: PATH lookup. | -| `MESHTASTIC_UHUBCTL_LOCATION_` | Pin a role to a specific uhubctl hub location (e.g. `1-1.3`). Wins over VID auto-detection — use when multiple devices share a VID. | +| `MESHTASTIC_UHUBCTL_LOCATION_` | Pin a role to a specific uhubctl hub location (e.g. `1-1.3`). Wins over VID auto-detection - use when multiple devices share a VID. | | `MESHTASTIC_UHUBCTL_PORT_` | Pin a role to a specific hub port number. Required alongside `LOCATION_`. | | `MESHTASTIC_UI_CAMERA_BACKEND` | Camera backend for UI tier + `capture_screen` tool: `opencv` / `ffmpeg` / `null` / `auto` (default). | | `MESHTASTIC_UI_CAMERA_DEVICE` | Generic camera device (index or path). Used by the UI tier when no per-role var is set. | diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000000..c7150cd2d07 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,23 @@ +# Claude Code instructions + +> **TL;DR** +> +> | | | +> | -------------- | ---------------------------------------------------------------------------------------------------------------------- | +> | Local tests | `./bin/run-tests.sh` (exit 0 GREEN · 1 RED · 2 AMBER · 3 FILTERED) | +> | Hardware tests | [meshtastic/meshtastic-mcp](https://github.com/meshtastic/meshtastic-mcp) (`MESHTASTIC_FIRMWARE_ROOT` → this checkout) | +> | Format | `trunk fmt` | +> | Mirror docs | `.github/copilot-instructions.md` (canonical) · `AGENTS.md` | +> +> **Need this? It's here.** +> +> | | | +> | ------------------------------------------- | ---------------------------------------------------------- | +> | General helpers (clamp, UTF-8, string fmt…) | `src/meshUtils.h` | +> | Logging macros (LOG_DEBUG / INFO / WARN…) | `src/DebugConfiguration.h` | +> | New module skeleton | inherit `ProtobufModule` in `src/mesh/ProtobufModule.h` | +> | Observer / event wiring | `src/Observer.h` | + +**Read `.github/copilot-instructions.md` first.** That file is the canonical agent-facing document for this repo. It covers project layout, coding conventions, the build system, CI/CD, the native C++ test suite, and the MCP Server & Hardware Test Harness. Read it top-to-bottom before starting any non-trivial change. + +This file (`CLAUDE.md`) is a short pointer for Claude Code sessions. Slash commands live in `.claude/commands/`. diff --git a/Dockerfile b/Dockerfile index ba013cb1557..c7b48313e50 100644 --- a/Dockerfile +++ b/Dockerfile @@ -14,7 +14,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1 RUN apt-get update && apt-get install --no-install-recommends -y \ curl wget g++ zip git ca-certificates pkg-config \ python3-pip python3-grpc-tools \ - libgpiod-dev libyaml-cpp-dev libbluetooth-dev libi2c-dev libuv1-dev \ + libgpiod-dev libyaml-cpp-dev libjsoncpp-dev libbluetooth-dev libi2c-dev libuv1-dev \ libusb-1.0-0-dev libulfius-dev liborcania-dev libssl-dev \ libx11-dev libinput-dev libxkbcommon-x11-dev libsqlite3-dev libsdl2-dev \ && apt-get clean && rm -rf /var/lib/apt/lists/* \ @@ -53,7 +53,7 @@ ENV TZ=Etc/UTC USER root RUN apt-get update && apt-get --no-install-recommends -y install \ - libc-bin libc6 libgpiod3 libyaml-cpp0.8 libi2c0 libuv1t64 libusb-1.0-0-dev \ + libc-bin libc6 libgpiod3 libyaml-cpp0.8 libjsoncpp26 libi2c0 libuv1t64 libusb-1.0-0-dev \ liborcania2.3 libulfius2.7t64 libssl3t64 \ libx11-6 libinput10 libxkbcommon-x11-0 libsdl2-2.0-0 \ && apt-get clean && rm -rf /var/lib/apt/lists/* \ diff --git a/Dockerfile.test b/Dockerfile.test index 12479b36d4e..a3077ea7240 100644 --- a/Dockerfile.test +++ b/Dockerfile.test @@ -7,7 +7,7 @@ ENV PIP_ROOT_USER_ACTION=ignore # hadolint ignore=DL3008 RUN apt-get update && apt-get install --no-install-recommends -y \ g++ git ca-certificates pkg-config \ - libgpiod-dev libyaml-cpp-dev libbluetooth-dev libi2c-dev libuv1-dev \ + libgpiod-dev libyaml-cpp-dev libjsoncpp-dev libbluetooth-dev libi2c-dev libuv1-dev \ libusb-1.0-0-dev libulfius-dev liborcania-dev libssl-dev \ libx11-dev libinput-dev libxkbcommon-x11-dev libsqlite3-dev libsdl2-dev \ && apt-get clean && rm -rf /var/lib/apt/lists/* \ diff --git a/README.md b/README.md index f34bf1839a0..de76f4bd3cf 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,8 @@ Meshtastic enables text messaging, location sharing, and telemetry over a decent ### Get Started -- 🔧 **[Building Instructions](https://meshtastic.org/docs/development/firmware/build)** – Learn how to compile the firmware from source. -- ⚡ **[Flashing Instructions](https://meshtastic.org/docs/getting-started/flashing-firmware/)** – Install or update the firmware on your device. +- 🔧 **[Building Instructions](https://meshtastic.org/docs/development/firmware/build)** - Learn how to compile the firmware from source. +- ⚡ **[Flashing Instructions](https://meshtastic.org/docs/getting-started/flashing-firmware/)** - Install or update the firmware on your device. Join our community and help improve Meshtastic! 🚀 diff --git a/SECURITY.md b/SECURITY.md index a77a4d028c4..34977e15a29 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -10,3 +10,35 @@ ## Reporting a Vulnerability We support the private reporting of potential security vulnerabilities. Please go to the Security tab to file a report with a description of the potential vulnerability and reproduction scripts (preferred) or steps, and our developers will review. + +Before filing, please read the Security Model below. Behavior whose only precondition is local API access to a node, or possession of a channel's pre-shared key, is intended by design and is not considered a vulnerability. + +## Security Model + +Meshtastic is an off-grid mesh protocol that runs on constrained microcontrollers within a 256 byte LoRa packet limit. These constraints shape its security design and rule out the heavier schemes used by IP-based protocols. This section summarizes what the firmware protects, the assumptions it rests on, and its known limits. Fuller write-ups are in the documentation: + +- Encryption overview: https://meshtastic.org/docs/overview/encryption/ +- Technical reference: https://meshtastic.org/docs/development/reference/encryption-technical/ +- Known limitations and future work: https://meshtastic.org/docs/about/overview/encryption/limitations/ + +### Cryptographic mechanisms + +- Channels are encrypted with a pre-shared key (PSK) using AES256-CTR. Channel traffic is encrypted but not authenticated, so anyone holding the PSK can read channel messages and can send messages as any node on that channel. +- Direct messages and admin messages use public key cryptography (x25519 key exchange with AES-CCM), providing confidentiality, authentication, and integrity between nodes on 2.5.0 or newer that have exchanged keys. +- Admin sessions use short-lived session IDs to limit replay of control messages. + +### Local trust boundary + +A client connected to a node over Bluetooth, USB serial, WiFi, or Ethernet has full local API access. From that connection it can read decrypted traffic, send messages as the node, change configuration (subject to managed mode), and read the node's private key for backup. This is intended behavior. The firmware trusts the local link the same way a phone or laptop trusts a directly attached device, and anything within reach of that connection (a shared LAN, a USB cable to an untrusted host, a paired phone) should be treated as part of the node itself. + +### Node identity (Trust On First Use) + +There is no central authority to sign node keys. The first public key a node hears for a given node number is the one it binds to that node number, a Trust On First Use (TOFU) model that is a hard requirement of a decentralized mesh. Clients and firmware reduce the impact of this by keeping favorited nodes from rolling out of the node database and by flagging public-key changes in the client UI. + +Firmware 2.8.X adds XEdDSA packet signing to further secure node identity claims and the authenticity of subsequent messages. It reuses each node's existing x25519 key pair to produce signatures, so a receiver can verify that a packet came from the holder of the bound key. Once a node has been seen signing, unsigned packets claiming that identity can be rejected. + +### Known limitations + +- No perfect forward secrecy. Traffic captured today can be decrypted later if a key is compromised, for example through a lost node or a mishandled channel key. +- Channel messages are not authenticated, as noted above. Although as of 2.8, channel messages will be xedDSA signed as a means of verification that is non-breaking. +- Setting WiFi credentials, or performing any other local administration, on an ESP32 over an untrusted network exposes that traffic, including the credentials, to the network. Provision and administer nodes over a trusted channel instead: Bluetooth, USB serial, or remote admin over the mesh. There is no current roadmap item to secure local administration over untrusted WiFi, though it may be addressed in a future release. diff --git a/alpine.Dockerfile b/alpine.Dockerfile index 6d1b999e299..facc180b8d5 100644 --- a/alpine.Dockerfile +++ b/alpine.Dockerfile @@ -4,7 +4,7 @@ # trunk-ignore-all(hadolint/DL3013): Do not pin pip package versions # Ensure the Alpine version is updated in both stages of the container! -FROM alpine:3.23 AS builder +FROM alpine:3.24 AS builder ARG PIO_ENV=native # Enable Alpine community repository (for 'py3-grpcio-tools') @@ -16,7 +16,7 @@ ENV PIP_BREAK_SYSTEM_PACKAGES=1 RUN apk --no-cache add \ bash g++ libstdc++-dev linux-headers zip git ca-certificates libbsd-dev \ py3-pip py3-grpcio-tools \ - libgpiod-dev yaml-cpp-dev bluez-dev \ + libgpiod-dev yaml-cpp-dev jsoncpp-dev bluez-dev \ libusb-dev i2c-tools-dev libuv-dev openssl-dev pkgconf argp-standalone \ libx11-dev libinput-dev libxkbcommon-dev sqlite-dev sdl2-dev \ && rm -rf /var/cache/apk/* \ @@ -35,7 +35,7 @@ RUN bash ./bin/build-native.sh "$PIO_ENV" && \ # ##### PRODUCTION BUILD ############# -FROM alpine:3.23 +FROM alpine:3.24 LABEL org.opencontainers.image.title="Meshtastic" \ org.opencontainers.image.description="Alpine Meshtastic daemon" \ org.opencontainers.image.url="https://meshtastic.org" \ @@ -48,7 +48,7 @@ LABEL org.opencontainers.image.title="Meshtastic" \ USER root RUN apk --no-cache add \ - shadow libstdc++ libbsd libgpiod yaml-cpp libusb \ + shadow libstdc++ libbsd libgpiod yaml-cpp jsoncpp libusb \ i2c-tools libuv libx11 libinput libxkbcommon sdl2 \ && rm -rf /var/cache/apk/* \ && mkdir -p /var/lib/meshtasticd \ diff --git a/bin/_rewrite_proto_namespace.py b/bin/_rewrite_proto_namespace.py new file mode 100755 index 00000000000..9c269f54335 --- /dev/null +++ b/bin/_rewrite_proto_namespace.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +"""Post-process protoc-generated Python files to live under a local namespace. + +Called by bin/regen-py-protos.sh. Walks the generated *_pb2.py files in the +target directory and rewrites every `meshtastic` reference (imports, dotted +attribute access) to use the new namespace (e.g., `meshtastic_v25`). + +Why: the .proto files declare `package meshtastic;`, so protoc emits +`from meshtastic import mesh_pb2 as ...` lines. That would shadow the PyPI +`meshtastic` package which the meshtastic-mcp tooling depends on. Renaming +to a local namespace keeps both available. + +Usage: + _rewrite_proto_namespace.py +""" + +from __future__ import annotations + +import pathlib +import re +import sys + + +def rewrite(dir_path: pathlib.Path, new_ns: str) -> int: + # Standard protoc import forms: + # from meshtastic.X_pb2 import ... (rare, for direct symbol pulls) + # from meshtastic import X_pb2 as ... (common, the cross-file ref) + # import meshtastic.X_pb2 (also possible) + pattern_dotted_from = re.compile(r"^from meshtastic\.", re.MULTILINE) + pattern_bare_from = re.compile(r"^from meshtastic import ", re.MULTILINE) + pattern_dotted_import = re.compile(r"^import meshtastic\.", re.MULTILINE) + + count = 0 + for p in dir_path.glob("*.py"): + text = p.read_text(encoding="utf-8") + new = pattern_dotted_from.sub(f"from {new_ns}.", text) + new = pattern_bare_from.sub(f"from {new_ns} import ", new) + new = pattern_dotted_import.sub(f"import {new_ns}.", new) + # NOTE: we deliberately leave `meshtastic/X.proto` source-filename + # references inside descriptor strings alone. The descriptor pool is + # keyed by source filename (independent of Python package layout), so + # those don't collide with the PyPI package's descriptors. + if new != text: + p.write_text(new, encoding="utf-8") + count += 1 + return count + + +def main(argv: list[str]) -> int: + if len(argv) != 2: + print("usage: _rewrite_proto_namespace.py ", file=sys.stderr) + return 2 + dir_path = pathlib.Path(argv[0]) + new_ns = argv[1] + if not dir_path.is_dir(): + print(f"directory not found: {dir_path}", file=sys.stderr) + return 2 + n = rewrite(dir_path, new_ns) + print(f"rewrote {n} file(s) in {dir_path} → namespace {new_ns}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bin/build-esp32.sh b/bin/build-esp32.sh index d07a09a1664..4e799b30a3a 100755 --- a/bin/build-esp32.sh +++ b/bin/build-esp32.sh @@ -38,4 +38,4 @@ cp bin/device-install.* $OUTDIR/ cp bin/device-update.* $OUTDIR/ echo "Copying manifest" -cp $BUILDDIR/$basename.mt.json $OUTDIR/$basename.mt.json || true +cp $BUILDDIR/$basename.mt.json $OUTDIR/$basename.mt.json diff --git a/bin/collect_sizes.py b/bin/collect_sizes.py new file mode 100644 index 00000000000..ad40a723f50 --- /dev/null +++ b/bin/collect_sizes.py @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 + +"""Collect firmware binary sizes from manifest (.mt.json) files into a single report. + +Output schema (consumed by bin/size_report.py): + {"": {"flash_bytes": , "ram_bytes": }} + +flash_bytes is the size of the main firmware image (.bin); for targets whose +packaged artifacts include no raw .bin (e.g. nRF52) it falls back to the +flash_bytes value (ELF text + data) emitted into the manifest by +bin/platformio-custom.py. ram_bytes is the static RAM footprint (.data + .bss) +emitted into the manifest by bin/platformio-custom.py; either metric is +omitted for manifests that predate it, and size_report.py renders those as +"n/a". +""" + +import json +import os +import sys + + +def collect_sizes(manifest_dir): + """Scan manifest_dir for .mt.json files and return {board: sizes_dict}.""" + sizes = {} + for fname in sorted(os.listdir(manifest_dir)): + if not fname.endswith(".mt.json"): + continue + path = os.path.join(manifest_dir, fname) + with open(path) as f: + data = json.load(f) + board = data.get("platformioTarget", fname.replace(".mt.json", "")) + # Find the main firmware .bin size (largest .bin, excluding OTA/littlefs/bleota) + bin_size = None + for entry in data.get("files", []): + name = entry.get("name", "") + if name.startswith("firmware-") and name.endswith(".bin"): + bin_size = entry["bytes"] + break + # Fallback: any .bin that isn't ota/littlefs/bleota + if bin_size is None: + for entry in data.get("files", []): + name = entry.get("name", "") + if name.endswith(".bin") and not any( + x in name for x in ["littlefs", "bleota", "ota"] + ): + bin_size = entry["bytes"] + break + # Fallback: flash footprint emitted into the manifest (ELF text + data), + # for targets that package no raw .bin (e.g. nRF52 hex/uf2/DFU zip) + if bin_size is None: + flash_bytes = data.get("flash_bytes") + if isinstance(flash_bytes, int) and not isinstance(flash_bytes, bool): + bin_size = flash_bytes + entry = {} + if bin_size is not None: + entry["flash_bytes"] = bin_size + ram_bytes = data.get("ram_bytes") + if isinstance(ram_bytes, int) and not isinstance(ram_bytes, bool): + entry["ram_bytes"] = ram_bytes + if entry: + sizes[board] = entry + return sizes + + +if __name__ == "__main__": + if len(sys.argv) != 3: + print(f"Usage: {sys.argv[0]} ", file=sys.stderr) + sys.exit(1) + + manifest_dir = sys.argv[1] + output_path = sys.argv[2] + + sizes = collect_sizes(manifest_dir) + with open(output_path, "w") as f: + json.dump(sizes, f, indent=2, sort_keys=True) + + print(f"Collected sizes for {len(sizes)} targets -> {output_path}") diff --git a/bin/config-dist.yaml b/bin/config-dist.yaml index 38fc057a000..46a044e8dd8 100644 --- a/bin/config-dist.yaml +++ b/bin/config-dist.yaml @@ -104,7 +104,9 @@ Lora: ### Define GPS GPS: -# SerialPath: /dev/ttyS0 +# SerialPath: /dev/ttyS0 # Use a physical serial GPS device (mutually exclusive with GpsdHost) +# GpsdHost: localhost # Use gpsd daemon as GPS source instead of a serial device +# GpsdPort: 2947 # gpsd TCP port (default: 2947) # ExtraPins: # - 22 @@ -151,6 +153,30 @@ Display: ### You can also specify the spi device for the display to use # spidev: spidev0.0 +### HUB75 RGB LED matrix panel (Raspberry Pi only, via hzeller/rpi-rgb-led-matrix). +### Requires the librgbmatrix library to be installed (so `pkg-config rgbmatrix` resolves) and +### the firmware built with it present. meshtasticd must run as root (or with /dev/gpiomem +### access) and on-board audio disabled (dtparam=audio=off) so it doesn't fight the PWM timing. +### The library owns the GPIO pins - they are chosen by HardwareMapping, not set individually. +# Panel: HUB75 +# HUB75: +# HardwareMapping: adafruit-hat-pwm # regular | adafruit-hat | adafruit-hat-pwm | regular-pi1 ... +# Rows: 64 # pixel rows per panel (e.g. 32 or 64) +# Cols: 64 # pixel columns per panel +# ChainLength: 1 # panels daisy-chained (width = Cols * ChainLength) +# Parallel: 1 # parallel chains (height = Rows * Parallel) +# Brightness: 80 # percent, 1..100 +# PWMBits: 11 # lower = higher refresh, fewer colors +# GPIOSlowdown: 4 # raise for faster Pis / long cables if the panel ghosts or flickers +# RGBSequence: RGB # reorder if your panel's colors are swapped (e.g. RBG, BGR) +# ScanMode: 0 # 0=progressive, 1=interlaced +# RowAddressType: 0 # 0=default, 1=AB-addressed (some 64x64 panels) +# Multiplexing: 0 # 0=direct, 1=stripe, 2=checker, ... +# PanelType: "" # e.g. FM6126A for panels needing a special init +# PixelMapper: "" # e.g. "U-mapper;Rotate:90" +# LimitRefreshRateHz: 0 # 0 = no limit +# DisableHardwarePulsing: false + Touchscreen: ### Note, at least for now, the touchscreen must have a CS pin defined, even if you let Linux manage the CS switching. @@ -165,12 +191,21 @@ Touchscreen: ### You can also specify the spi device for the touchscreen to use # spidev: spidev0.0 - Input: ### Configure device for direct keyboard input # KeyboardDevice: /dev/input/by-id/usb-_Raspberry_Pi_Internal_Keyboard-event-kbd +### Configure device for USB gamepad/joystick input (evdev). The D-pad drives +### Up/Down/Left/Right; buttons are mapped below. Find button codes with `evtest`. +# JoystickDevice: /dev/input/by-id/usb-0079_USB_Gamepad-event-joystick +### Map evdev button codes (hex or decimal) to actions. Omit to use built-in +### defaults (select=0x121, cancel=0x122). Actions: select, cancel, back, up, +### down, left, right, user. +# JoystickButtons: +# select: 0x122 +# cancel: 0x121 + ### Standard User Button Config # UserButton: 6 @@ -197,13 +232,11 @@ Webserver: # SSLKey: /etc/meshtasticd/ssl/private_key.pem # Path to SSL Key, generated if not present # SSLCert: /etc/meshtasticd/ssl/certificate.pem # Path to SSL Certificate, generated if not present - HostMetrics: # ReportInterval: 30 # Interval in minutes between HostMetrics report packets, or 0 for disabled # Channel: 0 # channel to send Host Metrics over. Defaults to the primary channel. # UserStringCommand: cat /sys/firmware/devicetree/base/serial-number # Command to execute, to send the results as the userString - Config: # DisplayMode: TWOCOLOR # uncomment to force BaseUI # DisplayMode: COLOR # uncomment to force MUI diff --git a/bin/config.d/display-hub75-64x64.yaml b/bin/config.d/display-hub75-64x64.yaml new file mode 100644 index 00000000000..704a4c6b9bb --- /dev/null +++ b/bin/config.d/display-hub75-64x64.yaml @@ -0,0 +1,37 @@ +Meta: + name: HUB75 RGB LED Matrix (64x64) + support: community + compatible: + - raspberry-pi + +### HUB75 RGB LED matrix panel driven by hzeller/rpi-rgb-led-matrix. +### +### Prerequisites: +### - Install the rpi-rgb-led-matrix library so `pkg-config rgbmatrix` resolves, and build the +### firmware with it present (the HUB75 backend is compiled in automatically when found). +### - meshtasticd must run as root (or have /dev/gpiomem access) for the library's direct GPIO. +### - Disable on-board audio (dtparam=audio=off in /boot/firmware/config.txt) - it shares the +### PWM hardware and causes flicker otherwise. +### +### The library owns the GPIO pins; they are selected by HardwareMapping (matching your HAT / +### wiring), not assigned individually. +Display: + Panel: HUB75 + HUB75: + HardwareMapping: adafruit-hat-pwm + Rows: 64 + Cols: 64 + ChainLength: 1 + Parallel: 1 + Brightness: 80 + PWMBits: 11 + GPIOSlowdown: 4 + RGBSequence: RGB + # Uncomment/tune as needed for your specific panel: + # ScanMode: 0 + # RowAddressType: 0 + # Multiplexing: 0 + # PanelType: FM6126A + # PixelMapper: "" + # LimitRefreshRateHz: 0 + # DisableHardwarePulsing: false diff --git a/bin/config.d/femtofox/femtofox_E80-900M2213S.yaml b/bin/config.d/femtofox/femtofox_E80-900M2213S.yaml new file mode 100644 index 00000000000..2f2b2460366 --- /dev/null +++ b/bin/config.d/femtofox/femtofox_E80-900M2213S.yaml @@ -0,0 +1,30 @@ +--- +Lora: +## Ebyte E80-900M22S +## This is a bit experimental +## +## + Module: lr1121 + gpiochip: 1 # subtract 32 from the gpio numbers + DIO3_TCXO_VOLTAGE: 1.8 + CS: 16 #pin6 / GPIO48 1C0 + IRQ: 23 #pin17 / GPIO55 1C7 + Busy: 22 #pin16 / GPIO54 1C6 + Reset: 25 #pin13 / GPIO57 1D1 + + + spidev: spidev0.0 #pins are (CS=16, CLK=17, MOSI=18, MISO=19) + spiSpeed: 2000000 + +rfswitch_table: + pins: [DIO5, DIO6, DIO7] + MODE_STBY: [LOW, LOW, LOW] + MODE_RX: [LOW, HIGH, LOW] + MODE_TX: [HIGH, HIGH, LOW] + MODE_TX_HP: [HIGH, LOW, LOW] + MODE_TX_HF: [LOW, LOW, LOW] + MODE_GNSS: [LOW, LOW, HIGH] + MODE_WIFI: [LOW, LOW, LOW] + +General: + MACAddressSource: eth0 diff --git a/bin/config.d/femtofox/femtofox_LR1121 generic.yaml b/bin/config.d/femtofox/femtofox_LR1121 generic.yaml new file mode 100644 index 00000000000..c66eebed542 --- /dev/null +++ b/bin/config.d/femtofox/femtofox_LR1121 generic.yaml @@ -0,0 +1,46 @@ +--- +Lora: +## Ebyte E80-900M22S +## This is a bit experimental +## +## + Module: lr1121 + gpiochip: 1 # subtract 32 from the gpio numbers + DIO3_TCXO_VOLTAGE: 1.8 + CS: 16 #pin6 / GPIO48 1C0 + IRQ: 23 #pin17 / GPIO55 1C7 + Busy: 22 #pin16 / GPIO54 1C6 + Reset: 25 #pin13 / GPIO57 1D1 + + + spidev: spidev0.0 #pins are (CS=16, CLK=17, MOSI=18, MISO=19) + spiSpeed: 2000000 + +rfswitch_table: + pins: + - DIO5 + - DIO6 + MODE_STBY: + - LOW + - LOW + MODE_RX: + - HIGH + - LOW + MODE_TX: + - HIGH + - HIGH + MODE_TX_HP: + - LOW + - HIGH + MODE_TX_HF: + - LOW + - LOW + MODE_GNSS: + - LOW + - LOW + MODE_WIFI: + - LOW + - LOW + +General: + MACAddressSource: eth0 diff --git a/bin/config.d/femtofox/femtofox_WIO-LR1121.yaml b/bin/config.d/femtofox/femtofox_WIO-LR1121.yaml new file mode 100644 index 00000000000..c2ab76d46dc --- /dev/null +++ b/bin/config.d/femtofox/femtofox_WIO-LR1121.yaml @@ -0,0 +1,30 @@ +--- +Lora: +## Ebyte E80-900M22S +## This is a bit experimental +## +## + Module: lr1121 + gpiochip: 1 # subtract 32 from the gpio numbers + DIO3_TCXO_VOLTAGE: 1.8 + CS: 16 #pin6 / GPIO48 1C0 + IRQ: 23 #pin17 / GPIO55 1C7 + Busy: 22 #pin16 / GPIO54 1C6 + Reset: 25 #pin13 / GPIO57 1D1 + + + spidev: spidev0.0 #pins are (CS=16, CLK=17, MOSI=18, MISO=19) + spiSpeed: 2000000 + +rfswitch_table: + pins: [DIO5, DIO6, DIO7] + MODE_STBY: [LOW, LOW, LOW] + MODE_RX: [LOW, LOW, LOW] + MODE_TX: [LOW, HIGH, LOW] + MODE_TX_HP: [HIGH, LOW, LOW] + # MODE_TX_HF: [] + # MODE_GNSS: [] + MODE_WIFI: [LOW, LOW, LOW] + +General: + MACAddressSource: eth0 diff --git a/bin/config.d/lora-FrameTastic-1262.yaml b/bin/config.d/lora-FrameTastic-1262.yaml new file mode 100644 index 00000000000..5acff775c17 --- /dev/null +++ b/bin/config.d/lora-FrameTastic-1262.yaml @@ -0,0 +1,20 @@ +#copy of lora-meshstick-1262.yaml data changed to match FrameTastic 1262 board + +Meta: + name: FrameTastic 1262 + support: community + compatible: + - usb + + +Lora: + Module: sx1262 + CS: 0 + IRQ: 6 + Reset: 2 + Busy: 4 + spidev: ch341 + DIO3_TCXO_VOLTAGE: true +# USB_Serialnum: 12345678 + USB_PID: 0x5512 + USB_VID: 0x1A86 diff --git a/bin/config.d/lora-NebraHat_1W.yaml b/bin/config.d/lora-NebraHat_1W.yaml new file mode 100644 index 00000000000..9f339371392 --- /dev/null +++ b/bin/config.d/lora-NebraHat_1W.yaml @@ -0,0 +1,19 @@ +# https://github.com/wehooper4/Meshtastic-Hardware/tree/main/NebraHat +# Use for 1 watt hat +Meta: + name: NebraHat 1W + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 # Nebra SX1262 Pi Hat - 1W + DIO2_AS_RF_SWITCH: true + DIO3_TCXO_VOLTAGE: true +# CS: 8 # Newer version of MeshtasticD do not need this? If issues uncomment this line + IRQ: 22 + Busy: 4 + Reset: 18 + RXen: 25 +I2C: + I2CDevice: /dev/i2c-1 \ No newline at end of file diff --git a/bin/config.d/lora-NebraHat_2W.yaml b/bin/config.d/lora-NebraHat_2W.yaml new file mode 100644 index 00000000000..4b712dd8a59 --- /dev/null +++ b/bin/config.d/lora-NebraHat_2W.yaml @@ -0,0 +1,20 @@ +# https://github.com/wehooper4/Meshtastic-Hardware/tree/main/NebraHat +# Use for 2 watt hat +Meta: + name: NebraHat 2W + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 # Nebra SX1262 Pi Hat - 2W + DIO2_AS_RF_SWITCH: true + DIO3_TCXO_VOLTAGE: true + SX126X_MAX_POWER: 8 +# CS: 8 # Newer version of MeshtasticD do not need this? If issues uncomment this line + IRQ: 22 + Busy: 4 + Reset: 18 + RXen: 25 +I2C: + I2CDevice: /dev/i2c-1 \ No newline at end of file diff --git a/bin/config.d/lora-PiTastic-1W.yaml b/bin/config.d/lora-PiTastic-1W.yaml new file mode 100644 index 00000000000..5c1bf95eef4 --- /dev/null +++ b/bin/config.d/lora-PiTastic-1W.yaml @@ -0,0 +1,20 @@ +#Board uses wehoopers zebra hat config # https://github.com/wehooper4/Meshtastic-Hardware/tree/main/ZebraHAT + +Meta: + name: PiTastic 1W + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 + DIO2_AS_RF_SWITCH: true + DIO3_TCXO_VOLTAGE: true + SX126X_MAX_POWER: 18 + CS: 24 + IRQ: 22 + Busy: 27 + Reset: 17 + +I2C: + I2CDevice: /dev/i2c-1 diff --git a/bin/config.d/lora-RAK6421-13300-slot1.yaml b/bin/config.d/lora-RAK6421-13300-slot1.yaml index a88544896d7..086dadbc554 100644 --- a/bin/config.d/lora-RAK6421-13300-slot1.yaml +++ b/bin/config.d/lora-RAK6421-13300-slot1.yaml @@ -18,4 +18,4 @@ Lora: DIO3_TCXO_VOLTAGE: true DIO2_AS_RF_SWITCH: true spidev: spidev0.0 - # CS: 8 \ No newline at end of file + # CS: 8 diff --git a/bin/config.d/lora-RAK6421-13300-slot2.yaml b/bin/config.d/lora-RAK6421-13300-slot2.yaml index 40b0cea095f..c0ed9017a09 100644 --- a/bin/config.d/lora-RAK6421-13300-slot2.yaml +++ b/bin/config.d/lora-RAK6421-13300-slot2.yaml @@ -5,7 +5,9 @@ Meta: - raspberry-pi Lora: - ### RAK13300 in Slot 2 pins + + ### RAK13300 in Slot 2 + Module: sx1262 IRQ: 18 #IO6 Reset: 24 # IO4 Busy: 19 # IO5 @@ -13,5 +15,7 @@ Lora: Enable_Pins: - 26 - 23 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 - # CS: 7 \ No newline at end of file + # CS: 7 diff --git a/bin/config.d/lora-RAK6421-13302-slot2.yaml b/bin/config.d/lora-RAK6421-13302-slot2.yaml index 5aa23911fe1..d4652c16747 100644 --- a/bin/config.d/lora-RAK6421-13302-slot2.yaml +++ b/bin/config.d/lora-RAK6421-13302-slot2.yaml @@ -5,14 +5,18 @@ Meta: - raspberry-pi Lora: - ### RAK13302 in Slot 2 pins + + ### RAK13302 in Slot 2 + Module: sx1262 IRQ: 18 #IO6 Reset: 24 # IO4 Busy: 19 # IO5 - # Ant_sw: 23 # IO3 + # Ant_sw: 23 # IO3 Enable_Pins: - 26 - 23 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: 7 TX_GAIN_LORA: [9, 9, 10, 11, 9, 8, 9, 10, 10, 10, 11, 12, 12, 12, 12, 12, 12, 12, 12, 10, 9, 8] \ No newline at end of file diff --git a/bin/config.d/lora-ZebraHat_1W.yaml b/bin/config.d/lora-ZebraHat_1W.yaml new file mode 100644 index 00000000000..f0b3c34d3fe --- /dev/null +++ b/bin/config.d/lora-ZebraHat_1W.yaml @@ -0,0 +1,19 @@ +# https://github.com/wehooper4/Meshtastic-Hardware/tree/main/ZebraHAT +# Use for 1 watt hat +Meta: + name: ZebraHat 1W + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 # Zebra SX1262 Pi Hat - 1W + DIO2_AS_RF_SWITCH: true + DIO3_TCXO_VOLTAGE: true + SX126X_MAX_POWER: 18 + CS: 24 + IRQ: 22 + Busy: 27 + Reset: 17 +I2C: + I2CDevice: /dev/i2c-1 diff --git a/bin/config.d/lora-ZebraHat_2W.yaml b/bin/config.d/lora-ZebraHat_2W.yaml new file mode 100644 index 00000000000..fe556ea77ad --- /dev/null +++ b/bin/config.d/lora-ZebraHat_2W.yaml @@ -0,0 +1,20 @@ +# https://github.com/wehooper4/Meshtastic-Hardware/tree/main/ZebraHAT +# Use for 2 watt hat +Meta: + name: ZebraHat 2W + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 # Zebra SX1262 Pi Hat - 2W + DIO2_AS_RF_SWITCH: true + DIO3_TCXO_VOLTAGE: true + SX126X_MAX_POWER: 8 + CS: 24 + IRQ: 22 + Busy: 27 + Reset: 17 + RXen: 25 +I2C: + I2CDevice: /dev/i2c-1 diff --git a/bin/config.d/lora-ecb41-pge-RAK6421-13300-slot2.yaml b/bin/config.d/lora-ecb41-pge-RAK6421-13300-slot2.yaml index e0ef946d97f..5121e8a61d5 100644 --- a/bin/config.d/lora-ecb41-pge-RAK6421-13300-slot2.yaml +++ b/bin/config.d/lora-ecb41-pge-RAK6421-13300-slot2.yaml @@ -29,6 +29,8 @@ Lora: - pin: 50 # GPIO1_C2 (physical 16) gpiochip: 1 line: 18 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: # GPIO0_A7 (SPI1_CSN1, physical 26) # pin: 7 diff --git a/bin/config.d/lora-ecb41-pge-RAK6421-13302-slot2.yaml b/bin/config.d/lora-ecb41-pge-RAK6421-13302-slot2.yaml index 5548bc5c7b5..0417d36e90e 100644 --- a/bin/config.d/lora-ecb41-pge-RAK6421-13302-slot2.yaml +++ b/bin/config.d/lora-ecb41-pge-RAK6421-13302-slot2.yaml @@ -29,6 +29,8 @@ Lora: - pin: 50 # GPIO1_C2 (physical 16) gpiochip: 1 line: 18 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: # GPIO0_A7 (SPI1_CSN1, physical 26) # pin: 7 diff --git a/bin/config.d/lora-lyra-zero-BQ-SG3-BQ35LORA900V1M-slot1.yaml b/bin/config.d/lora-lyra-zero-BQ-SG3-BQ35LORA900V1M-slot1.yaml new file mode 100644 index 00000000000..0bee549e459 --- /dev/null +++ b/bin/config.d/lora-lyra-zero-BQ-SG3-BQ35LORA900V1M-slot1.yaml @@ -0,0 +1,29 @@ +# Station G3 + BQ35LORA900V1M Primary Slot +# Board Doc: https://wiki.bqvoy.com/en/devkits/station-g3 +Meta: + name: B&Q Station G3 + BQ35LORA900V1M Primary Slot + support: official + compatible: + - luckfox-lyra-zero-w # Armbian + +Lora: + Module: sx1262 + IRQ: # GPIO0_A5 (physical 15) + pin: 5 + gpiochip: 0 + line: 5 + Reset: # GPIO1_D1 (physical 36) + pin: 57 + gpiochip: 1 + line: 25 + Busy: # GPIO0_B4 (physical 18) + pin: 12 + gpiochip: 0 + line: 12 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true + spidev: spidev0.0 + # CS: # GPIO0_B2 (physical 24) + # pin: 10 + # gpiochip: 0 + # line: 10 diff --git a/bin/config.d/lora-lyra-zero-RAK6421-13300-slot2.yaml b/bin/config.d/lora-lyra-zero-RAK6421-13300-slot2.yaml index 255a3eca38b..63648432c47 100644 --- a/bin/config.d/lora-lyra-zero-RAK6421-13300-slot2.yaml +++ b/bin/config.d/lora-lyra-zero-RAK6421-13300-slot2.yaml @@ -29,6 +29,8 @@ Lora: - pin: 13 # GPIO0_B5 gpiochip: 0 line: 13 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: # GPIO0_B1 # pin: 9 diff --git a/bin/config.d/lora-lyra-zero-RAK6421-13302-slot2.yaml b/bin/config.d/lora-lyra-zero-RAK6421-13302-slot2.yaml index 773a35ab0fe..d3baebd458c 100644 --- a/bin/config.d/lora-lyra-zero-RAK6421-13302-slot2.yaml +++ b/bin/config.d/lora-lyra-zero-RAK6421-13302-slot2.yaml @@ -29,6 +29,8 @@ Lora: - pin: 13 # GPIO0_B5 gpiochip: 0 line: 13 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: # GPIO0_B1 # pin: 9 diff --git a/bin/config.d/lora-ok3506-RAK6421-13300-slot2.yaml b/bin/config.d/lora-ok3506-RAK6421-13300-slot2.yaml index 969c20ad3f4..225fb5f8dd9 100644 --- a/bin/config.d/lora-ok3506-RAK6421-13300-slot2.yaml +++ b/bin/config.d/lora-ok3506-RAK6421-13300-slot2.yaml @@ -31,6 +31,8 @@ Lora: - pin: 103 # GPIO3_A7 (physical 16) gpiochip: 3 line: 7 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: # GPIO0_B7 (SPI0_CSN1, physical 26) # pin: 15 diff --git a/bin/config.d/lora-ok3506-RAK6421-13302-slot2.yaml b/bin/config.d/lora-ok3506-RAK6421-13302-slot2.yaml index 36b70658b6a..ffb221fba1f 100644 --- a/bin/config.d/lora-ok3506-RAK6421-13302-slot2.yaml +++ b/bin/config.d/lora-ok3506-RAK6421-13302-slot2.yaml @@ -31,6 +31,8 @@ Lora: - pin: 103 # GPIO3_A7 (physical 16) gpiochip: 3 line: 7 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true spidev: spidev0.1 # CS: # GPIO0_B7 (SPI0_CSN1, physical 26) # pin: 15 diff --git a/bin/config.d/lora-piggystick-lr1121.yaml b/bin/config.d/lora-piggystick-lr1121.yaml index e11c78dd3eb..d6b88198b05 100644 --- a/bin/config.d/lora-piggystick-lr1121.yaml +++ b/bin/config.d/lora-piggystick-lr1121.yaml @@ -1,5 +1,5 @@ Meta: - name: Lora Meshstick SX1262 + name: PiggyStick LR1121 support: community compatible: - usb @@ -12,6 +12,6 @@ Lora: Busy: 4 spidev: ch341 DIO3_TCXO_VOLTAGE: 1.8 -# USB_Serialnum: 12345678 + # USB_Serialnum: 12345678 USB_PID: 0x5512 USB_VID: 0x1A86 diff --git a/bin/config.d/lora-pimesh-1w-v1.yaml b/bin/config.d/lora-pimesh-1w-v1.yaml new file mode 100644 index 00000000000..ceb33b5cd94 --- /dev/null +++ b/bin/config.d/lora-pimesh-1w-v1.yaml @@ -0,0 +1,21 @@ +# PiMesh-1W (V1) E22-900M30S +# https://meshsmith.net/products/pimesh-1w +Meta: + name: PiMesh-1W (V1) E22-900M30S + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 + CS: 21 + IRQ: 16 + Reset: 18 + Busy: 20 + TXen: 13 + RXen: 12 + DIO3_TCXO_VOLTAGE: true + SX126X_MAX_POWER: 22 + spidev: spidev0.0 + +# preamble_length: 17 diff --git a/bin/config.d/lora-pimesh-1w-v2.yaml b/bin/config.d/lora-pimesh-1w-v2.yaml new file mode 100644 index 00000000000..b6b812f3206 --- /dev/null +++ b/bin/config.d/lora-pimesh-1w-v2.yaml @@ -0,0 +1,21 @@ +# PiMesh-1W (V2) E22P-915M30S / E22P-868M30S +# https://meshsmith.net/products/pimesh-1w +Meta: + name: PiMesh-1W (V2) E22P-915M30S / E22P-868M30S + support: community + compatible: + - raspberry-pi + +Lora: + Module: sx1262 + IRQ: 6 + Reset: 18 + Busy: 5 + CS: 8 + DIO2_AS_RF_SWITCH: true + DIO3_TCXO_VOLTAGE: true + SX126X_MAX_POWER: 22 + spidev: spidev0.0 + +# add the following to /boot/firmware/config.txt +# gpio=26=op,dh diff --git a/bin/config.d/lora-rpi-BQ-SG3-BQ35LORA900V1M-slot1.yaml b/bin/config.d/lora-rpi-BQ-SG3-BQ35LORA900V1M-slot1.yaml new file mode 100644 index 00000000000..d58889517d7 --- /dev/null +++ b/bin/config.d/lora-rpi-BQ-SG3-BQ35LORA900V1M-slot1.yaml @@ -0,0 +1,17 @@ +# Station G3 + BQ35LORA900V1M Primary Slot +# Board Doc: https://wiki.bqvoy.com/en/devkits/station-g3 +Meta: + name: B&Q Station G3 + BQ35LORA900V1M Primary Slot + support: official + compatible: + - raspberry-pi + +Lora: + Module: sx1262 + IRQ: 22 + Reset: 16 + Busy: 24 + DIO3_TCXO_VOLTAGE: true + DIO2_AS_RF_SWITCH: true + spidev: spidev0.0 + #CS: 8 diff --git a/bin/config.d/lora-usb-meshtoad-e22.yaml b/bin/config.d/lora-usb-meshtoad-e22.yaml index 49182c83e2f..3d8b15783c6 100644 --- a/bin/config.d/lora-usb-meshtoad-e22.yaml +++ b/bin/config.d/lora-usb-meshtoad-e22.yaml @@ -1,3 +1,5 @@ +# This config works with all revisions of the Meshtoad USB radio. + Meta: name: meshtoad-e22 support: official diff --git a/bin/device-install.sh b/bin/device-install.sh index 52a848c3829..3d20376bce4 100755 --- a/bin/device-install.sh +++ b/bin/device-install.sh @@ -7,9 +7,9 @@ MCU="" # Constants RESET_BAUD=1200 FIRMWARE_OFFSET=0x00 -# Default littlefs* offset. -OFFSET=0x300000 -# Default OTA Offset +# Fallback offsets, used only when firmware metadata carries no partition table. +# These track partition-table.csv; boards with their own table override them below. +SPIFFS_OFFSET=0x300000 OTA_OFFSET=0x260000 # Determine the correct esptool command to use @@ -122,8 +122,15 @@ if [[ -f "$FILENAME" && "$FILENAME" == *.factory.bin ]]; then jq . "$METAFILE" # Extract relevant fields from metadata if [[ $(jq -r '.part' "$METAFILE") != "null" ]]; then - OTA_OFFSET=$(jq -r '.part[] | select(.subtype == "ota_1") | .offset' "$METAFILE") - SPIFFS_OFFSET=$(jq -r '.part[] | select(.subtype == "spiffs") | .offset' "$METAFILE") + META_OTA_OFFSET=$(jq -r '.part[] | select(.subtype == "ota_1") | .offset' "$METAFILE") + META_SPIFFS_OFFSET=$(jq -r '.part[] | select(.subtype == "spiffs") | .offset' "$METAFILE") + # A partition table may omit either entry; keep the built-in default then. + if [[ -n "$META_OTA_OFFSET" && "$META_OTA_OFFSET" != "null" ]]; then + OTA_OFFSET="$META_OTA_OFFSET" + fi + if [[ -n "$META_SPIFFS_OFFSET" && "$META_SPIFFS_OFFSET" != "null" ]]; then + SPIFFS_OFFSET="$META_SPIFFS_OFFSET" + fi fi MCU=$(jq -r '.mcu' "$METAFILE") else @@ -154,9 +161,9 @@ if [[ -f "$FILENAME" && "$FILENAME" == *.factory.bin ]]; then $ESPTOOL_CMD ${ESPTOOL_ERASE_FLASH} $ESPTOOL_CMD ${ESPTOOL_WRITE_FLASH} $FIRMWARE_OFFSET "${FILENAME}" echo "Trying to flash ${OTAFILE} at offset ${OTA_OFFSET}" - $ESPTOOL_CMD ${ESPTOOL_WRITE_FLASH} $OTA_OFFSET "${OTAFILE}" - echo "Trying to flash ${SPIFFSFILE}, at offset ${OFFSET}" - $ESPTOOL_CMD ${ESPTOOL_WRITE_FLASH} $OFFSET "${SPIFFSFILE}" + $ESPTOOL_CMD ${ESPTOOL_WRITE_FLASH} "$OTA_OFFSET" "${OTAFILE}" + echo "Trying to flash ${SPIFFSFILE}, at offset ${SPIFFS_OFFSET}" + $ESPTOOL_CMD ${ESPTOOL_WRITE_FLASH} "$SPIFFS_OFFSET" "${SPIFFSFILE}" else show_help diff --git a/bin/eth-ota-upload.py b/bin/eth-ota-upload.py new file mode 100644 index 00000000000..68bc1e2a7fd --- /dev/null +++ b/bin/eth-ota-upload.py @@ -0,0 +1,252 @@ +#!/usr/bin/env python3 +""" +Meshtastic Ethernet OTA Upload Tool + +Uploads firmware to RP2350-based Meshtastic devices via Ethernet (W5500). +Compresses firmware with GZIP and sends it over TCP using the MOTA protocol. +Authenticates using SHA256 challenge-response with a pre-shared key (PSK). + +Usage: + python bin/eth-ota-upload.py --host 192.168.1.100 firmware.bin + python bin/eth-ota-upload.py --host 192.168.1.100 --psk mySecretKey firmware.bin + python bin/eth-ota-upload.py --host 192.168.1.100 --psk-hex 6d65736874... firmware.bin +""" + +import argparse +import gzip +import hashlib +import socket +import struct +import sys +import time + +# Default PSK matching the firmware default: "meshtastic_ota_default_psk_v1!!!" +DEFAULT_PSK = b"meshtastic_ota_default_psk_v1!!!" + + +def crc32(data: bytes) -> int: + """Compute CRC32 matching ErriezCRC32 (standard CRC32 with final XOR).""" + import binascii + + return binascii.crc32(data) & 0xFFFFFFFF + + +def load_firmware(path: str) -> bytes: + """Load firmware file, compressing with GZIP if not already compressed.""" + # Reject UF2 files — OTA requires raw .bin firmware + if path.lower().endswith(".uf2"): + bin_path = path.rsplit(".", 1)[0] + ".bin" + print(f"ERROR: UF2 files cannot be used for OTA updates.") + print(f" The Updater/picoOTA expects raw .bin firmware.") + print(f" Try: {bin_path}") + sys.exit(1) + + with open(path, "rb") as f: + data = f.read() + + # Check if already GZIP compressed (magic bytes 1f 8b) + if data[:2] == b"\x1f\x8b": + print(f"Firmware already GZIP compressed: {len(data):,} bytes") + return data + + print(f"Firmware raw size: {len(data):,} bytes") + compressed = gzip.compress(data, compresslevel=9) + ratio = len(compressed) / len(data) * 100 + print(f"GZIP compressed: {len(compressed):,} bytes ({ratio:.1f}%)") + return compressed + + +def authenticate(sock: socket.socket, psk: bytes) -> bool: + """Perform SHA256 challenge-response authentication with the device.""" + # Receive 32-byte nonce from server + nonce = b"" + while len(nonce) < 32: + chunk = sock.recv(32 - len(nonce)) + if not chunk: + print("ERROR: Connection closed during authentication") + return False + nonce += chunk + + # Compute SHA256(nonce || PSK) + h = hashlib.sha256() + h.update(nonce) + h.update(psk) + response = h.digest() + + # Send 32-byte response + sock.sendall(response) + + # Wait for auth result (1 byte) + result = sock.recv(1) + if not result: + print("ERROR: No authentication response") + return False + + if result[0] == 0x06: # ACK + print("Authentication successful.") + return True + elif result[0] == 0x07: # OTA_ERR_AUTH + print("ERROR: Authentication failed — wrong PSK") + return False + else: + print(f"ERROR: Unexpected auth response 0x{result[0]:02X}") + return False + + +def upload_firmware(host: str, port: int, firmware: bytes, psk: bytes, timeout: float) -> bool: + """Upload firmware over TCP using the MOTA protocol with PSK authentication.""" + fw_crc = crc32(firmware) + fw_size = len(firmware) + + print(f"Connecting to {host}:{port}...") + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.settimeout(timeout) + + try: + sock.connect((host, port)) + print("Connected.") + + # Step 1: Authenticate + print("Authenticating...") + if not authenticate(sock, psk): + return False + + # Step 2: Send 12-byte MOTA header: magic(4) + size(4) + crc32(4) + header = struct.pack("<4sII", b"MOTA", fw_size, fw_crc) + sock.sendall(header) + print(f"Header sent: size={fw_size:,}, CRC32=0x{fw_crc:08X}") + + # Wait for ACK (1 byte) + ack = sock.recv(1) + if not ack or ack[0] != 0x06: + error_codes = { + 0x02: "Size error", + 0x04: "Invalid magic", + 0x05: "Update.begin() failed", + } + code = ack[0] if ack else 0xFF + msg = error_codes.get(code, f"Unknown error 0x{code:02X}") + print(f"ERROR: Server rejected header: {msg}") + return False + + print("Header accepted. Uploading firmware...") + + # Send firmware in 1KB chunks + chunk_size = 1024 + sent = 0 + start_time = time.time() + + while sent < fw_size: + end = min(sent + chunk_size, fw_size) + chunk = firmware[sent:end] + sock.sendall(chunk) + sent = end + + # Progress bar + pct = sent * 100 // fw_size + bar_len = 40 + filled = bar_len * sent // fw_size + bar = "█" * filled + "░" * (bar_len - filled) + elapsed = time.time() - start_time + speed = sent / elapsed if elapsed > 0 else 0 + sys.stdout.write(f"\r [{bar}] {pct:3d}% {sent:,}/{fw_size:,} ({speed/1024:.1f} KB/s)") + sys.stdout.flush() + + elapsed = time.time() - start_time + print(f"\n Transfer complete in {elapsed:.1f}s") + + # Wait for final result (1 byte) + print("Waiting for verification...") + result = sock.recv(1) + if not result: + print("ERROR: No response from device") + return False + + result_codes = { + 0x00: "OK — Update staged, device rebooting", + 0x01: "CRC mismatch", + 0x02: "Size error", + 0x03: "Write error", + 0x04: "Magic mismatch", + 0x05: "Updater.begin() failed", + 0x07: "Auth failed", + 0x08: "Timeout", + } + code = result[0] + msg = result_codes.get(code, f"Unknown result 0x{code:02X}") + + if code == 0x00: + print(f"SUCCESS: {msg}") + return True + else: + print(f"ERROR: {msg}") + return False + + except socket.timeout: + print("ERROR: Connection timed out") + return False + except ConnectionRefusedError: + print(f"ERROR: Connection refused by {host}:{port}") + return False + except OSError as e: + print(f"ERROR: {e}") + return False + finally: + sock.close() + + +def main(): + parser = argparse.ArgumentParser( + description="Upload firmware to Meshtastic RP2350 devices via Ethernet OTA" + ) + parser.add_argument("firmware", help="Path to firmware .bin or .bin.gz file") + parser.add_argument("--host", required=True, help="Device IP address") + parser.add_argument( + "--port", type=int, default=4243, help="OTA port (default: 4243)" + ) + parser.add_argument( + "--timeout", + type=float, + default=60.0, + help="Socket timeout in seconds (default: 60)", + ) + psk_group = parser.add_mutually_exclusive_group() + psk_group.add_argument( + "--psk", + type=str, + help="Pre-shared key as UTF-8 string (default: meshtastic_ota_default_psk_v1!!!)", + ) + psk_group.add_argument( + "--psk-hex", + type=str, + help="Pre-shared key as hex string (e.g., 6d65736874...)", + ) + args = parser.parse_args() + + # Resolve PSK + if args.psk: + psk = args.psk.encode("utf-8") + elif args.psk_hex: + try: + psk = bytes.fromhex(args.psk_hex) + except ValueError: + print("ERROR: Invalid hex string for --psk-hex") + sys.exit(1) + else: + psk = DEFAULT_PSK + + print("Meshtastic Ethernet OTA Upload") + print("=" * 40) + + firmware = load_firmware(args.firmware) + + if upload_firmware(args.host, args.port, firmware, psk, args.timeout): + print("\nDevice is rebooting with new firmware.") + sys.exit(0) + else: + print("\nUpload failed.") + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/bin/gen-fake-nodedb-seed.py b/bin/gen-fake-nodedb-seed.py new file mode 100755 index 00000000000..d8cf3f4b96e --- /dev/null +++ b/bin/gen-fake-nodedb-seed.py @@ -0,0 +1,439 @@ +#!/usr/bin/env python3 +"""Deterministic seed-data generator for the fake NodeDB fixture pipeline. + +Writes a JSONL file describing N fake-but-realistic Meshtastic peers. +The output is hand-editable and committed; a sibling compile step +(bin/seed-json-to-proto.py) turns it into a binary `meshtastic_NodeDatabase` +v25 protobuf with fresh "now-relative" timestamps. + +Determinism contract: + Same --seed -> byte-identical JSONL output, regardless of wall clock. + All timestamps are stored as `*_offset_sec` (seconds before "now"); the + compile step resolves them to absolute epochs at compile time. + +Structural fields covered: + * NodeInfoLite header: num, long_name, short_name, hw_model, role, + public_key, snr, channel, hops_away, next_hop, bitfield flags + * PositionLite: lat/long Gaussian around --centroid, altitude, source + * DeviceMetrics: battery/voltage/util/uptime + * EnvironmentMetrics: temp/humidity/pressure/iaq + * StatusMessage: error_code (usually zero) + +Active-board allow-list: + hw_model values are restricted to the intersection of + (a) variants with `custom_meshtastic_support_level = 1` in + variants/*/*/platformio.ini, AND + (b) values present in the `HardwareModel` enum in mesh.proto. + See HW_MODEL_WEIGHTS below. Deprecated boards (legacy TLORA / Heltec V1-2 / + classic TBEAM / TBEAM_V0P7 / Nano G1 / etc.) and fuzzer-only sentinels + (PORTDUINO, ANDROID_SIM, DIY_V1, ...) are excluded. + +Active-role allow-list: + Excludes ROUTER_CLIENT (deprecated v2.3.15) and REPEATER (deprecated v2.7.11). +""" + +from __future__ import annotations + +import argparse +import datetime as _dt +import json +import math +import pathlib +import random +import sys + +# -------------------------------------------------------------------------- +# Active-board allow-list (intersection of tier-1 variants + HardwareModel enum). +# Refresh by running: +# for f in $(find variants -name 'platformio.ini' | xargs grep -lE 'custom_meshtastic_support_level = 1'); do +# grep custom_meshtastic_hw_model_slug $f | awk -F= '{print $2}' | tr -d ' '; +# done | sort -u | comm -12 - <(python3 -c "from meshtastic.protobuf.mesh_pb2 import HardwareModel; print('\\n'.join(HardwareModel.keys()))" | sort) +# -------------------------------------------------------------------------- +HW_MODEL_WEIGHTS: dict[str, float] = { + "HELTEC_V3": 14.0, + "T_DECK": 9.0, + "HELTEC_V4": 8.0, + "RAK4631": 8.0, + "HELTEC_MESH_POCKET": 6.0, + "TRACKER_T1000_E": 5.0, + "HELTEC_MESH_NODE_T114": 5.0, + "T_DECK_PRO": 5.0, + "LILYGO_TBEAM_S3_CORE": 4.0, + "HELTEC_WIRELESS_PAPER": 4.0, + "HELTEC_WSL_V3": 3.0, + "T_ECHO": 3.0, + "HELTEC_WIRELESS_TRACKER": 3.0, + "HELTEC_WIRELESS_TRACKER_V2": 2.0, + "HELTEC_VISION_MASTER_E290": 2.0, + "HELTEC_MESH_SOLAR": 2.0, + "SEEED_WIO_TRACKER_L1": 2.0, + "T_LORA_PAGER": 1.5, + "HELTEC_VISION_MASTER_E213": 1.5, + "T_ECHO_PLUS": 1.0, + "MUZI_BASE": 1.0, + "WISMESH_TAP_V2": 1.0, + "THINKNODE_M2": 1.0, + "THINKNODE_M5": 1.0, + "TLORA_T3_S3": 1.0, + # Long tail (uniform low weight across remaining tier-1 boards): + "HELTEC_V4_R8": 0.3, + "HELTEC_VISION_MASTER_T190": 0.3, + "HELTEC_HT62": 0.3, + "HELTEC_MESH_NODE_T096": 0.3, + "M5STACK_C6L": 0.3, + "MINI_EPAPER_S3": 0.3, + "MUZI_R1_NEO": 0.3, + "NOMADSTAR_METEOR_PRO": 0.3, + "RAK3312": 0.3, + "RAK3401": 0.3, + "SEEED_SOLAR_NODE": 0.3, + "SEEED_WIO_TRACKER_L1_EINK": 0.3, + "SENSECAP_INDICATOR": 0.3, + "TBEAM_1_WATT": 0.3, + "THINKNODE_M1": 0.3, + "THINKNODE_M3": 0.3, + "THINKNODE_M6": 0.3, + "T_ECHO_LITE": 0.3, + "WISMESH_TAG": 0.3, + "WISMESH_TAP": 0.3, + "XIAO_NRF52_KIT": 0.3, + "CROWPANEL": 0.3, +} + +# Non-deprecated roles only. +ROLE_WEIGHTS: dict[str, float] = { + "CLIENT": 75.0, + "CLIENT_MUTE": 5.0, + "ROUTER": 7.0, + "TRACKER": 3.0, + "SENSOR": 2.0, + "CLIENT_HIDDEN": 2.0, + "ROUTER_LATE": 2.0, + "CLIENT_BASE": 2.0, + "TAK": 1.0, + "TAK_TRACKER": 0.5, + "LOST_AND_FOUND": 0.5, +} + +# Name pools — 60 firsts × 60 lasts = 3600 combinations. +FIRSTS = [ + "Quick", "Brave", "Silent", "Wild", "Lone", "Bright", "Red", "Blue", + "Green", "Black", "White", "Iron", "Steel", "Copper", "Silver", "Gold", + "Stone", "River", "Forest", "Mountain", "Canyon", "Desert", "Storm", "Sky", + "Solar", "Lunar", "Dawn", "Dusk", "Misty", "Frosty", "Sunny", "Shady", + "Happy", "Sleepy", "Drowsy", "Sneaky", "Sharp", "Smooth", "Rough", "Loud", + "Soft", "Slow", "Fast", "Tall", "Short", "Old", "New", "Tiny", + "Giant", "Hidden", "Lost", "Found", "Wandering", "Roving", "Drifting", "Floating", + "Burning", "Frozen", "Whispering", "Howling", +] +LASTS = [ + "Phoenix", "Lion", "Bear", "Wolf", "Hawk", "Eagle", "Fox", "Lynx", + "Cougar", "Coyote", "Raven", "Owl", "Crow", "Falcon", "Heron", "Crane", + "Otter", "Badger", "Bison", "Elk", "Moose", "Stag", "Doe", "Hare", + "Marmot", "Mole", "Beaver", "Squirrel", "Mustang", "Bronco", "Pony", "Colt", + "Cobra", "Viper", "Mamba", "Adder", "Gecko", "Iguana", "Tortoise", "Turtle", + "Salmon", "Trout", "Bass", "Pike", "Shark", "Whale", "Dolphin", "Seal", + "Cactus", "Yucca", "Sage", "Juniper", "Pine", "Cedar", "Aspen", "Oak", + "Bluff", "Mesa", "Arroyo", "Ridge", +] + +# Brief callsign pool for licensed-looking suffixes. +CALLSIGN_PREFIXES = ["KX", "WD", "N5", "KE", "AB", "W5", "K1", "KQ", "AE", "NM"] + +# Only emojis that fit in 4 UTF-8 bytes (no variation selectors). short_name's +# nanopb max_size:5 (incl. NUL) limits content to 4 bytes. ❄️ / ☀️ would be +# 6 bytes due to U+FE0F variation selector — explicitly excluded. +EMOJI_SHORTNAMES = ["🦊", "🐺", "🦅", "🐢", "🌵", "🔥", "🌙", + "🌊", "🗻", "🌲", "🦌", "🐝", "🦂", "🦉", + "🦇", "🦋"] + +# -------------------------------------------------------------------------- +# Helpers +# -------------------------------------------------------------------------- + +NUM_RESERVED = 4 # firmware reserves 0..3 (per NodeDB constants) +NUM_MAX_EXCLUSIVE = 0x80000000 # restrict to positive int32 range for readability + + +def _weighted_choice(rng: random.Random, weights: dict[str, float]) -> str: + """Deterministic weighted pick. Uses sorted keys so dict order is fixed.""" + keys = sorted(weights.keys()) + totals = [weights[k] for k in keys] + return rng.choices(keys, weights=totals, k=1)[0] + + +def _gen_long_name(rng: random.Random, is_licensed: bool) -> str: + base = f"{rng.choice(FIRSTS)} {rng.choice(LASTS)}" + if is_licensed: + prefix = rng.choice(CALLSIGN_PREFIXES) + # Two trailing alpha chars after the digit; keep within 25 - len(base) - 1 + suffix = f" {prefix}{rng.randint(0,9)}{rng.choice('ABCDEFGHIJKLMNOPQRSTUVWXYZ')}{rng.choice('ABCDEFGHIJKLMNOPQRSTUVWXYZ')}" + # nanopb max_size:25 means C string fits 24 bytes + NUL. + if len(base) + len(suffix) <= 24: + base = base + suffix + # Hard cap to 24 chars (nanopb max_size:25 minus NUL). + return base[:24] + + +def _gen_short_name(rng: random.Random, long_name: str) -> str: + # 10% emoji-only short_name + if rng.random() < 0.10: + return rng.choice(EMOJI_SHORTNAMES) + first_char = long_name[0].upper() if long_name else "X" + alphanums = "ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789" + return first_char + "".join(rng.choices(alphanums, k=3)) + + +def _gen_hops_away(rng: random.Random) -> int: + # Geometric-ish: 0→55%, 1→25%, 2→12%, 3→5%, 4→2%, 5+→1% + r = rng.random() + if r < 0.55: + return 0 + if r < 0.80: + return 1 + if r < 0.92: + return 2 + if r < 0.97: + return 3 + if r < 0.99: + return 4 + return rng.randint(5, 7) + + +def _gen_position( + rng: random.Random, + centroid_lat: float, + centroid_lon: float, + spread_km: float, + last_heard_offset_sec: int, +) -> dict: + # 1 deg ≈ 111 km at the equator; we use this as a flat approximation. + lat = centroid_lat + rng.gauss(0.0, spread_km / 111.0) + lon = centroid_lon + rng.gauss(0.0, spread_km / 111.0) + altitude = max(0, round(rng.gauss(1376.0, 250.0))) # T or C valley floor + relief + # Position was reported up to 300s before last_heard. + time_offset_sec = last_heard_offset_sec + rng.randint(0, 300) + return { + "latitude": round(lat, 6), + "longitude": round(lon, 6), + "altitude": altitude, + "time_offset_sec": time_offset_sec, + "location_source": "LOC_INTERNAL", + } + + +def _gen_telemetry(rng: random.Random) -> dict: + # 5% plugged-in (battery_level == 101); rest uniform [10..100]. + if rng.random() < 0.05: + battery_level = 101 + voltage = 4.20 + else: + battery_level = rng.randint(10, 100) + voltage = round(3.3 + (battery_level / 100.0) * 0.9, 3) + # Beta distributions for low/right-skewed metrics; randomly draw via gammavariate. + def _beta(a: float, b: float) -> float: + x = rng.gammavariate(a, 1.0) + y = rng.gammavariate(b, 1.0) + return x / (x + y) + channel_utilization = round(_beta(2.0, 15.0) * 100.0, 2) + air_util_tx = round(_beta(1.5, 20.0) * 10.0, 3) + uptime_seconds = int(rng.expovariate(1.0 / 86400.0)) + return { + "battery_level": battery_level, + "voltage": voltage, + "channel_utilization": channel_utilization, + "air_util_tx": air_util_tx, + "uptime_seconds": uptime_seconds, + } + + +def _gen_environment(rng: random.Random) -> dict: + return { + "temperature": round(rng.gauss(22.0, 8.0), 2), + "relative_humidity": round(min(100.0, max(0.0, rng.gauss(55.0, 20.0))), 2), + "barometric_pressure": round(rng.gauss(1013.0, 8.0), 2), + "iaq": int(min(500, max(0, round(rng.gauss(50.0, 30.0))))), + } + + +def _gen_status(rng: random.Random) -> dict: + # `StatusMessage` (mesh.proto:1445) has a single free-form `string status`. + # Most peers report a healthy short status; occasional alert string. + healthy = ["OK", "online", "active", "running", "ready", "nominal"] + alert = ["low-batt", "no-gps", "weak-signal", "rebooted", "offline-soon"] + if rng.random() < 0.92: + return {"status": rng.choice(healthy)} + return {"status": rng.choice(alert)} + + +def _gen_node( + rng: random.Random, + num: int, + centroid_lat: float, + centroid_lon: float, + spread_km: float, + coverage: dict[str, float], + last_heard_mean_sec: int, + last_heard_max_sec: int, +) -> dict: + is_licensed = rng.random() < 0.05 + long_name = _gen_long_name(rng, is_licensed) + short_name = _gen_short_name(rng, long_name) + hw_model = _weighted_choice(rng, HW_MODEL_WEIGHTS) + role = _weighted_choice(rng, ROLE_WEIGHTS) + has_public_key = rng.random() < 0.92 + public_key_hex = ( + "".join(f"{rng.randint(0,255):02x}" for _ in range(32)) if has_public_key else "" + ) + snr = round(max(-20.0, min(12.0, rng.gauss(6.0, 4.0))), 2) + channel = 0 if rng.random() < 0.90 else rng.randint(1, 7) + hops_away = _gen_hops_away(rng) + next_hop = rng.randint(0, 255) if hops_away > 0 else 0 + last_heard_offset_sec = int(min(rng.expovariate(1.0 / last_heard_mean_sec), last_heard_max_sec)) + + bitfield = { + "has_user": True, + "is_favorite": rng.random() < 0.08, + "is_muted": rng.random() < 0.03, + "via_mqtt": rng.random() < 0.12, + "is_ignored": rng.random() < 0.01, + "is_licensed": is_licensed, + "has_is_unmessagable": True, + "is_unmessagable": rng.random() < 0.02, + "is_key_manually_verified": rng.random() < 0.04, + } + + node: dict = { + "num": f"0x{num:08x}", + "long_name": long_name, + "short_name": short_name, + "hw_model": hw_model, + "role": role, + "public_key_hex": public_key_hex, + "snr": snr, + "channel": channel, + "hops_away": hops_away, + "next_hop": next_hop, + "last_heard_offset_sec": last_heard_offset_sec, + "bitfield": bitfield, + "position": ( + _gen_position(rng, centroid_lat, centroid_lon, spread_km, last_heard_offset_sec) + if rng.random() < coverage["position"] + else None + ), + "telemetry": _gen_telemetry(rng) if rng.random() < coverage["telemetry"] else None, + "environment": _gen_environment(rng) if rng.random() < coverage["environment"] else None, + "status": _gen_status(rng) if rng.random() < coverage["status"] else None, + } + return node + + +def _parse_my_node_num(s: str | None) -> int | None: + if s is None: + return None + s = s.strip() + if s.startswith("0x") or s.startswith("0X"): + return int(s, 16) + return int(s) + + +def main(argv: list[str]) -> int: + p = argparse.ArgumentParser( + description="Deterministic JSONL seed for the fake NodeDB fixture.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + p.add_argument("--count", type=int, required=True, help="Number of fake nodes to emit.") + p.add_argument("--seed", type=int, required=True, help="Deterministic seed.") + p.add_argument("--out", required=True, help="Output JSONL path.") + p.add_argument( + "--centroid", + default="33.1284,-107.2528", + help="LAT,LON centroid (default: Truth or Consequences, NM).", + ) + p.add_argument("--spread-km", type=float, default=60.0, help="Gaussian std-dev in km.") + p.add_argument("--position-coverage", type=float, default=0.85) + p.add_argument("--telemetry-coverage", type=float, default=0.70) + p.add_argument("--environment-coverage", type=float, default=0.25) + p.add_argument("--status-coverage", type=float, default=0.40) + p.add_argument("--my-node-num", default=None, help="Exclude this NodeNum from generated set (hex or dec).") + p.add_argument("--last-heard-mean-sec", type=int, default=3600) + p.add_argument("--last-heard-max-sec", type=int, default=7 * 86400) + args = p.parse_args(argv) + + if args.count <= 0: + print("--count must be positive", file=sys.stderr) + return 2 + + try: + centroid_lat, centroid_lon = (float(s) for s in args.centroid.split(",")) + except ValueError: + print(f"--centroid must be LAT,LON; got {args.centroid!r}", file=sys.stderr) + return 2 + + my_node_num = _parse_my_node_num(args.my_node_num) + + rng = random.Random(args.seed) + + # 1) Generate a unique deterministic set of NodeNums. + nums: set[int] = set() + while len(nums) < args.count: + n = rng.randrange(NUM_RESERVED, NUM_MAX_EXCLUSIVE) + if my_node_num is not None and n == my_node_num: + continue + nums.add(n) + ordered_nums = sorted(nums) # sort to fix output order independent of set hash + + # 2) Per-node generation (in num order, single RNG continues). + coverage = { + "position": args.position_coverage, + "telemetry": args.telemetry_coverage, + "environment": args.environment_coverage, + "status": args.status_coverage, + } + nodes = [ + _gen_node( + rng, + n, + centroid_lat, + centroid_lon, + args.spread_km, + coverage, + args.last_heard_mean_sec, + args.last_heard_max_sec, + ) + for n in ordered_nums + ] + + # 3) Write JSONL. + out_path = pathlib.Path(args.out) + out_path.parent.mkdir(parents=True, exist_ok=True) + # `generated_at_iso` is informational; it does NOT affect determinism because + # we derive it from the seed, not from wall clock. (Same seed -> same string.) + generated_at = _dt.datetime.fromtimestamp(args.seed, tz=_dt.timezone.utc).isoformat().replace("+00:00", "Z") + meta = { + "_meta": { + "version": 25, + "seed": args.seed, + "count": args.count, + "centroid": [centroid_lat, centroid_lon], + "spread_km": args.spread_km, + "generated_at_iso": generated_at, + "my_node_num_excluded": (None if my_node_num is None else f"0x{my_node_num:08x}"), + "coverage": coverage, + "last_heard_mean_sec": args.last_heard_mean_sec, + "last_heard_max_sec": args.last_heard_max_sec, + } + } + with out_path.open("w", encoding="utf-8") as f: + # `ensure_ascii=False` so emoji short_names survive. `sort_keys=True` for + # determinism (insertion order varies by Python version otherwise). + f.write(json.dumps(meta, ensure_ascii=False, sort_keys=True) + "\n") + for node in nodes: + f.write(json.dumps(node, ensure_ascii=False, sort_keys=True) + "\n") + + print(f"wrote {args.count} nodes to {out_path} ({out_path.stat().st_size} bytes)", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bin/lint-ifdef-complexity.sh b/bin/lint-ifdef-complexity.sh new file mode 100755 index 00000000000..b42c7d068f0 --- /dev/null +++ b/bin/lint-ifdef-complexity.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# lint-ifdef-complexity.sh - flag preprocessor conditionals that OR together +# too many defined() terms (e.g. long display-driver chains in main.cpp). +# +# Such chains should be replaced by a single umbrella feature macro computed +# once in a central header (this repo already has HAS_TFT / HAS_SCREEN in +# src/configuration.h). The compiler has no warning for this, so we lint it. +# +# Emits one line per offending logical #if/#elif in the format +# ::::: +# which trunk parses via parse_regex. Always exits 0; findings go to stdout. +# +# Usage: lint-ifdef-complexity.sh [ ...] +# Tune the limit with MAX_DEFINED (default 5 → warns at 6+ terms). + +set -euo pipefail + +MAX_DEFINED="${MAX_DEFINED:-5}" + +awk -v max="${MAX_DEFINED}" ' + # Start of a #if / #elif (but not #ifdef / #ifndef, which carry one term). + /^[[:space:]]*#[[:space:]]*(if|elif)([[:space:](]|$)/ { + startln = FNR + logical = $0 + # Splice backslash line-continuations into one logical line. + while (logical ~ /\\[[:space:]]*$/ && (getline nxt) > 0) { + sub(/\\[[:space:]]*$/, "", logical) + logical = logical " " nxt + } + tmp = logical + n = gsub(/defined/, "", tmp) # gsub returns the match count + if (n > max) { + printf "%s:%d:1:warning:preprocessor conditional ORs %d defined() terms (limit %d); replace the driver list with an umbrella feature macro such as HAS_TFT or HAS_SCREEN:too-many-defined\n", \ + FILENAME, startln, n, max + } + } +' "$@" diff --git a/bin/platformio-custom.py b/bin/platformio-custom.py index b75c666241c..08d010e086d 100644 --- a/bin/platformio-custom.py +++ b/bin/platformio-custom.py @@ -45,6 +45,78 @@ def infer_architecture(board_cfg): return "stm32" return None +def run_size_tool(env, flag, purpose): + """Run the toolchain size tool against the built ELF and return its output. + + Shared plumbing for compute_ram_bytes / compute_flash_bytes. Returns None + when the ELF is missing or the tool fails; the manifest then simply omits + the metric and downstream size reports show "n/a". + """ + elf = env.File(env.subst("$BUILD_DIR/${PROGNAME}.elf")) + if not elf.exists(): + return None + size_tool = env.subst("$SIZETOOL") or "size" + try: + return subprocess.check_output( + [size_tool, flag, elf.get_abspath()], + env=env["ENV"], + universal_newlines=True, + ) + except Exception as exc: + print(f"mtjson: skipping {purpose} ({size_tool} failed: {exc})") + return None + +def compute_ram_bytes(env): + """Static RAM usage (.data + .bss) of the ELF, via the toolchain size tool. + + Deliberately excludes heap/stack placeholder sections (e.g. the nRF52 .heap + section): on nRF52840 the heap arena is the linker gap after .bss, so static + RAM growth shrinks the usable heap 1:1 - which is exactly why we track it. + Returns None when the value cannot be determined; the manifest then simply + omits ram_bytes and downstream size reports show "n/a". + """ + output = run_size_tool(env, "-A", "ram_bytes") + if output is None: + return None + ram = 0 + found = False + for line in output.splitlines(): + parts = line.split() + if len(parts) < 2: + continue + name = parts[0] + # Main-SRAM static sections: .data/.bss, platform-prefixed variants (e.g. + # ESP32 .dram0.data/.dram0.bss), and RISC-V small-data .sdata/.sbss. + # ESP-IDF .rtc.* sections live outside the heap-competing SRAM; .heap and + # .tdata never match. + if name.startswith(".rtc"): + continue + if name in (".data", ".bss", ".sdata", ".sbss") or name.endswith(".data") or name.endswith(".bss"): + try: + ram += int(parts[1]) + found = True + except ValueError: + continue + return ram if found else None + +def compute_flash_bytes(env): + """Flash footprint (text + data) of the ELF via the toolchain size tool. + + Approximates the flashed app image for targets whose packaged artifacts do + not include a raw .bin (e.g. nRF52: hex/uf2/DFU zip only); consumed by + bin/collect_sizes.py as a fallback flash measurement for the size report + and the bin/ram_budgets.json budget gate. Returns None when the value + cannot be determined; the manifest then simply omits flash_bytes. + """ + output = run_size_tool(env, "-B", "flash_bytes") + if output is None: + return None + for line in output.splitlines(): + parts = line.split() + if len(parts) >= 3 and parts[0].isdigit() and parts[1].isdigit(): + return int(parts[0]) + int(parts[1]) + return None + def manifest_gather(source, target, env): global manifest_ran if manifest_ran: @@ -98,9 +170,9 @@ def manifest_gather(source, target, env): d["part_name"] = partition_map[p] out.append(d) print(d) - manifest_write(out, env) + manifest_write(out, env, compute_ram_bytes(env), compute_flash_bytes(env)) -def manifest_write(files, env): +def manifest_write(files, env, ram_bytes=None, flash_bytes=None): # Defensive: also skip manifest writing if we cannot determine architecture def get_project_option(name): try: @@ -137,6 +209,14 @@ def as_list(val): "has_mui": False, "has_inkhud": False, } + # Static RAM footprint (.data + .bss); consumed by bin/collect_sizes.py for + # the CI size report and the bin/ram_budgets.json budget gate. + if ram_bytes is not None: + manifest["ram_bytes"] = ram_bytes + # Flash footprint (text + data); fallback flash measurement for targets + # whose packaged artifacts include no raw .bin (e.g. nRF52). + if flash_bytes is not None: + manifest["flash_bytes"] = flash_bytes # Get partition table (generated in esp32_pre.py) if it exists if env.get("custom_mtjson_part"): # custom_mtjson_part is a JSON string, convert it back to a dict @@ -260,6 +340,18 @@ def as_list(val): lb.env.Append(CPPDEFINES=[("APP_VERSION", verObj["long"])]) break +# DEBUG_MUTE already compiles every LOG_* call out entirely; nanopb's own error-message +# strings are a separate mechanism it doesn't touch, so mirror the same intent into nanopb. +debug_mute_defined = any( + d == "DEBUG_MUTE" or (isinstance(d, tuple) and d[0] == "DEBUG_MUTE") for d in env.get("CPPDEFINES", []) +) +if debug_mute_defined: + projenv.Append(CPPDEFINES=[("PB_NO_ERRMSG", 1)]) + for lb in env.GetLibBuilders(): + if lb.name == "Nanopb": + lb.env.Append(CPPDEFINES=[("PB_NO_ERRMSG", 1)]) + break + # Get the display resolution from macros def get_display_resolution(build_flags): # Check "DISPLAY_SIZE" to determine the screen resolution @@ -293,9 +385,12 @@ def load_boot_logo(source, target, env): board_arch = infer_architecture(env.BoardConfig()) should_skip_manifest = board_arch is None -# For host/native envs, avoid depending on 'buildprog' (some targets don't define it) -mtjson_deps = [] if should_skip_manifest else ["buildprog"] -if not should_skip_manifest and platform.name == "espressif32": +# Most platforms can generate the manifest as part of the default 'buildprog' target. +# Typically this passes success/failure properly. +mtjson_deps = ["buildprog"] +if platform.name == "espressif32": + # On ESP32, we need to explicitly depend upon the binary to prevent fake-success upon failure. + mtjson_deps = ["$BUILD_DIR/${PROGNAME}.bin"] # Build littlefs image as part of mtjson target # Equivalent to `pio run -t buildfs` target_lfs = env.DataToBin( @@ -309,7 +404,8 @@ def skip_manifest(source, target, env): env.AddCustomTarget( name="mtjson", - dependencies=mtjson_deps, + # For host/native envs, avoid depending on 'buildprog' (some targets don't define it) + dependencies=[], actions=[skip_manifest], title="Meshtastic Manifest (skipped)", description="mtjson generation is skipped for native environments", diff --git a/bin/ram_budgets.json b/bin/ram_budgets.json new file mode 100644 index 00000000000..b48903a0b82 --- /dev/null +++ b/bin/ram_budgets.json @@ -0,0 +1,24 @@ +{ + "_comment": [ + "Per-environment static-size budgets, enforced by the size-budget-gate CI job", + "via: bin/size_report.py --budgets bin/ram_budgets.json --enforce-budgets", + "Only environments listed here are gated, and only for the metrics they list.", + "", + "ram_bytes = static RAM (.data + .bss from the ELF, emitted into the .mt.json", + "manifest by bin/platformio-custom.py). On nRF52840 the heap arena is the linker", + "gap after .bss, so every byte of static RAM growth shrinks the usable heap 1:1;", + "that is how the 2.8.0 heap regression shipped without CI noticing.", + "", + "flash_bytes = size of the main firmware image (.bin). The rak4631 app region is", + "0x27000..0xEA000 = 798,720 bytes, and the image must also stay clear of the", + "warm-store record-ring guard (extra_scripts/nrf52_warm_region.py).", + "", + "Budgets are raised DELIBERATELY, never automatically: if your change needs more", + "headroom, bump the limit here in the same PR and justify the increase in the PR", + "description." + ], + "rak4631": { + "ram_bytes": 113000, + "flash_bytes": 786000 + } +} diff --git a/bin/regen-fake-nodedbs.sh b/bin/regen-fake-nodedbs.sh new file mode 100755 index 00000000000..b9d6d477519 --- /dev/null +++ b/bin/regen-fake-nodedbs.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Regenerate the fake-NodeDB fixtures: produces 250 / 500 / 1000 / 2000-node +# JSONL seed files + their compiled v25 protobufs. +# +# Layout: +# test/fixtures/nodedb/seed_v25_.jsonl — COMMITTED, hand-editable. +# build/fixtures/nodedb/nodes_v25_.proto — .gitignored, build artifact. +# Drop into /prefs/nodes.proto. +# +# Daily use: ./bin/regen-fake-nodedbs.sh +# - Recompiles protos from committed seeds (fresh wall-clock timestamps). +# Intentional seed bump: REGEN_SEEDS=yes ./bin/regen-fake-nodedbs.sh +# - Overwrites the committed JSONL files with freshly-seeded data. + +set -euo pipefail +cd "$(dirname "$0")/.." + +# 1) Make sure the Python protobuf bindings exist (in-tree generation; .gitignored). +if [[ ! -d bin/_generated/meshtastic ]]; then + echo "regenerating Python protobuf bindings (one-time)..." + ./bin/regen-py-protos.sh +fi + +# 2) Pick a Python interpreter that has the meshtastic deps installed. +# Prefer a local venv; otherwise fall back to system python3 (install the +# `meshtastic` package there, or run this under `uv run --with meshtastic`). +PY="python3" +for cand in .venv/bin/python3; do + if [[ -x "$cand" ]]; then + PY="$cand" + break + fi +done + +# 3) Pinned seeds per size — bump only when you intentionally want different +# structural data committed. Parallel arrays so the script works on +# macOS bash 3.2 (no `declare -A`). +SIZES=(250 500 1000 2000) +SEEDS=(20260511 20260512 20260513 20260514) + +REGEN_SEEDS="${REGEN_SEEDS:-no}" + +mkdir -p build/fixtures/nodedb test/fixtures/nodedb + +for i in 0 1 2 3; do + n="${SIZES[$i]}" + seed="${SEEDS[$i]}" + jsonl=$(printf "test/fixtures/nodedb/seed_v25_%04d.jsonl" "$n") + proto=$(printf "build/fixtures/nodedb/nodes_v25_%04d.proto" "$n") + + if [[ "$REGEN_SEEDS" == "yes" || ! -f "$jsonl" ]]; then + $PY bin/gen-fake-nodedb-seed.py \ + --count "$n" \ + --seed "$seed" \ + --out "$jsonl" \ + --centroid 33.1284,-107.2528 \ + --spread-km 60 \ + --position-coverage 0.85 \ + --telemetry-coverage 0.70 \ + --environment-coverage 0.25 \ + --status-coverage 0.40 + echo " seed: $jsonl ($(wc -c < "$jsonl") bytes)" + fi + + $PY bin/seed-json-to-proto.py --in "$jsonl" --out "$proto" + echo " proto: $proto ($(wc -c < "$proto") bytes)" +done + +echo "" +echo "Done. To load on Portduino native:" +echo " cp build/fixtures/nodedb/nodes_v25_1000.proto ~/.portduino/default/prefs/nodes.proto" +echo "" +echo "To push to a hardware device:" +echo " Use the meshtastic-mcp tool: push_fake_nodedb(size=1000, target=\"hardware\", port=\"/dev/cu.usbmodemXXXX\", confirm=True)" +echo " (MCP server: https://github.com/meshtastic/meshtastic-mcp)" diff --git a/bin/regen-py-protos.sh b/bin/regen-py-protos.sh new file mode 100755 index 00000000000..c5e73f0fc01 --- /dev/null +++ b/bin/regen-py-protos.sh @@ -0,0 +1,51 @@ +#!/usr/bin/env bash +# Regenerate Python protobuf bindings from the in-tree `protobufs/` submodule +# into `bin/_generated/`. Called by bin/regen-fake-nodedbs.sh; also useful as +# a standalone refresh after any change to a .proto file. +# +# Output is .gitignored — bindings are a build artifact. +# +# Namespace rewrite: +# The .proto files declare `package meshtastic;`, which makes protoc emit +# imports like `from meshtastic import mesh_pb2`. That conflicts with the +# PyPI `meshtastic` package (which the meshtastic-mcp tooling relies on for its +# SerialInterface/BLEInterface transport). We post-process the generated +# files to live under `meshtastic_v25` instead — both the directory layout +# and all internal imports — so they coexist cleanly with the PyPI package. + +set -euo pipefail +cd "$(dirname "$0")/.." + +if ! command -v protoc >/dev/null 2>&1; then + echo "ERROR: protoc not found in PATH." >&2 + echo " macOS: brew install protobuf" >&2 + echo " Ubuntu/Debian: apt install protobuf-compiler" >&2 + exit 1 +fi + +OUT=bin/_generated +LOCAL_NS=meshtastic_v25 + +rm -rf "$OUT" +mkdir -p "$OUT" + +# 1) Generate from the in-tree protos. nanopb.proto first so its descriptor +# is available for the [(nanopb).*] options on other messages. +protoc \ + --proto_path=protobufs \ + --python_out="$OUT" \ + protobufs/nanopb.proto \ + protobufs/meshtastic/*.proto + +# 2) Move the generated `meshtastic/` directory to `meshtastic_v25/`. +mv "$OUT/meshtastic" "$OUT/$LOCAL_NS" + +# 3) Rewrite internal imports: any reference to `meshtastic.X_pb2` or +# `from meshtastic import X_pb2` becomes `meshtastic_v25.*`. +python3 bin/_rewrite_proto_namespace.py "$OUT/$LOCAL_NS" "$LOCAL_NS" + +# 4) Make the package importable. +touch "$OUT/__init__.py" +touch "$OUT/$LOCAL_NS/__init__.py" + +echo "regenerated Python protobuf bindings -> $OUT/$LOCAL_NS/ (namespace: $LOCAL_NS)" >&2 diff --git a/bin/run-tests.sh b/bin/run-tests.sh new file mode 100755 index 00000000000..de824ad822b --- /dev/null +++ b/bin/run-tests.sh @@ -0,0 +1,311 @@ +#!/usr/bin/env bash +# Run native PlatformIO unit tests and emit a single, unambiguous verdict. +# +# Why this exists: PlatformIO reports failures three different ways ([FAILED], :FAIL:, +# [ERRORED]) and an all-pass run prints "N succeeded" with NO "0 failed" clause — so naive +# greps produce false greens (see .notes/test-passfail-filter.md). This script encodes the +# correct logic once, and cross-checks the number of suites that actually ran against the +# canonical set in test/ so a suite silently going missing shows up as AMBER, not green. +# +# Usage: +# ./bin/run-tests.sh # run all suites, full verdict + count cross-check +# ./bin/run-tests.sh -f test_utf8 # run one suite (yields FILTERED, not GREEN) +# ./bin/run-tests.sh -e native # override env (default: coverage) +# ./bin/run-tests.sh --quiet # only print the final RESULT line +# +# Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER, 3 = FILTERED. +# +# Verdicts: +# GREEN — all canonical suites ran, all passed, no ignored test cases. +# AMBER — all that ran passed, but something was lost: a suite silently went missing on a +# full run, or individual test cases were skipped (Unity TEST_IGNORE / :IGNORE:). +# FILTERED — a -f run completed cleanly; suites not in the filter were intentionally skipped. +# Use this when iterating on a single suite; it is not a quality signal. +# RED — at least one failure, build error, or sanitizer fault. +# +# The final line is machine-readable, e.g.: +# RESULT: GREEN N/N suites passed +# RESULT: AMBER N/M suites ran (missing: test_radio test_serial) — all that ran passed +# RESULT: AMBER 3 test case(s) ignored +# RESULT: FILTERED 1/N suites ran (not run: …) — filtered: test_utf8 +# RESULT: RED test_traffic_management: 1 failed (or: build/crash error) +# RESULT: RED sanitizer fault — SUMMARY: AddressSanitizer: 1272 byte(s) leaked (tests may have +# all passed; the coverage build aborts at exit on an ASan/LSan fault — often shown only +# as [ERRORED]/SIGHUP. The script names it and points at running the binary bare.) + +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +ROOT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)" +cd "$ROOT_DIR" + +ENV="coverage" +FILTER="" +QUIET=false +PASSTHRU=() + +while [[ $# -gt 0 ]]; do + case "$1" in + -f) + FILTER="$2" + PASSTHRU+=("-f" "$2") + shift 2 + ;; + -e) + ENV="$2" + shift 2 + ;; + --quiet) + QUIET=true + shift + ;; + *) + PASSTHRU+=("$1") + shift + ;; + esac +done + +# Locate pio (PATH, then the standard PlatformIO venv). +PIO="$(command -v pio || command -v platformio || echo "$HOME/.platformio/penv/bin/pio")" +if [[ ! -x $PIO ]] && ! command -v "$PIO" >/dev/null 2>&1; then + echo "RESULT: RED pio not found (looked in PATH and ~/.platformio/penv/bin)" + exit 1 +fi + +LOG="$(mktemp -t meshtest.XXXXXX.log)" +MARKER="" +PROGRESS_PID="" +trap 'rm -f "$LOG" "${MARKER:-}"; [[ -n ${PROGRESS_PID:-} ]] && kill "$PROGRESS_PID" 2>/dev/null' EXIT + +# Canonical suite set = the directories in test/. This is the source of truth for +# "what should run"; a filtered run only expects its filtered suite. +mapfile -t ALL_SUITES < <(find test -maxdepth 1 -type d -name 'test_*' -printf '%f\n' | sort) +EXPECTED_COUNT=${#ALL_SUITES[@]} + +# Canonical suite count — the registered total, maintained in test/native-suite-count. +# Update that file whenever a test suite is added or removed. +CANONICAL_COUNT_FILE="test/native-suite-count" +if [[ -f $CANONICAL_COUNT_FILE ]]; then + CANONICAL_COUNT=$(tr -d '[:space:]' <"$CANONICAL_COUNT_FILE") +else + CANONICAL_COUNT="" +fi + +# Cached object-count for this env, written after each completed build (in the gitignored build +# dir). Used as the progress denominator: accurate for a full rebuild (every object recompiles), +# only a rough upper bound for an incremental run. +BASELINE_FILE=".pio/build/${ENV}/.runtests-objcount" + +# Progress trail file (gitignored build dir). ALWAYS written so a backgrounded/piped run can be +# checked mid-build with `tail -f` — that's the whole point: don't fly blind on a 20-min rebuild. +PROGRESS_FILE=".pio/build/${ENV}/.runtests-progress" + +# --- Progress heartbeat ------------------------------------------------------ +# Emit ONE status line every few seconds: build = objects (re)compiled this run / cached total + +# best-effort ETA; test = suites finished / expected. Appends to $PROGRESS_FILE always (tail it to +# check on a backgrounded run); also live-updates the tty when $5=1 (interactive --quiet). Never +# touches $LOG, which is parsed for the verdict, so piped/CI captures stay clean. +progress_monitor() { + local marker="$1" objtotal="$2" testtotal="$3" pfile="$4" totty="$5" start now el done ran eta line + start=$(date +%s) + while :; do + now=$(date +%s) + el=$((now - start)) + if grep -q 'Testing\.\.\.' "$LOG" 2>/dev/null; then + ran=$(grep -cE "${ENV}:test_[a-z0-9_]+ \[(PASSED|FAILED|ERRORED)\]" "$LOG" 2>/dev/null) + line=$(printf '[test] %s/%s suites done — %dm%02ds' "$ran" "$testtotal" $((el / 60)) $((el % 60))) + else + done=$(find ".pio/build/${ENV}" -name '*.o' -newer "$marker" 2>/dev/null | wc -l) + if ((objtotal > 0 && done > 0)); then + eta=$((objtotal > done ? (objtotal - done) * el / done : 0)) + line=$(printf '[build] %d/%d objs — %dm%02ds — ETA ~%dm%02ds' \ + "$done" "$objtotal" $((el / 60)) $((el % 60)) $((eta / 60)) $((eta % 60))) + else + # done==0 (incremental: nothing to rebuild yet) or no cached baseline — no ETA yet. + line=$(printf '[build] %d objs compiled — %dm%02ds' "$done" $((el / 60)) $((el % 60))) + fi + fi + printf '%s\n' "$line" >>"$pfile" 2>/dev/null # file trail (always) + [[ $totty == 1 ]] && printf '\r\033[K%s' "$line" >/dev/tty 2>/dev/null # live line (human) + sleep 4 + done +} + +# Launch the heartbeat for every run. It writes the progress file unconditionally; the live tty +# line only when interactive AND --quiet (where pio's own output is hidden — otherwise pio's +# streamed compile lines already show progress and a \r line would just fight them). +mkdir -p ".pio/build/${ENV}" 2>/dev/null || true +: >"$PROGRESS_FILE" 2>/dev/null || true +MARKER="$(mktemp -t meshtest-mark.XXXXXX)" +TOTTY=0 +{ $QUIET && [[ -t 1 ]]; } && TOTTY=1 +progress_monitor "$MARKER" "$(cat "$BASELINE_FILE" 2>/dev/null || echo 0)" \ + "$([[ -n $FILTER ]] && echo 1 || echo "$EXPECTED_COUNT")" "$PROGRESS_FILE" "$TOTTY" & +PROGRESS_PID=$! + +if ! $QUIET; then + echo "Running: $PIO test -e $ENV ${PASSTHRU[*]-} (expecting $EXPECTED_COUNT suites)" +fi +echo "progress: tail -f $PROGRESS_FILE" >&2 +if [[ ! -t 1 ]] && ! $QUIET; then + echo "hint: stdout is a pipe — build errors appear at the top of output and may be lost; use --quiet to get just the RESULT line" >&2 +fi + +# Run pio, tee to log. PIPESTATUS[0] is pio's real exit (NOT tee's). +if $QUIET; then + "$PIO" test -e "$ENV" "${PASSTHRU[@]}" >"$LOG" 2>&1 +else + "$PIO" test -e "$ENV" "${PASSTHRU[@]}" 2>&1 | tee "$LOG" +fi +PIO_RC=${PIPESTATUS[0]} + +# Stop the heartbeat, clear its line, and cache this build's object total for next time. +if [[ -n $PROGRESS_PID ]]; then + kill "$PROGRESS_PID" 2>/dev/null + wait "$PROGRESS_PID" 2>/dev/null + PROGRESS_PID="" + # Clear the live line only if we were writing one — opening /dev/tty when there is none is + # itself a redirect-open error the trailing 2>/dev/null cannot suppress. + [[ $TOTTY == 1 ]] && printf '\r\033[K' >/dev/tty 2>/dev/null +fi +[[ -d ".pio/build/${ENV}" ]] && find ".pio/build/${ENV}" -name '*.o' 2>/dev/null | wc -l >"$BASELINE_FILE" 2>/dev/null || true + +# --- Outcome detection ------------------------------------------------------- +# The SAME outcome is spelled differently depending on which layer emitted the line — this is +# the trap that produces false greens (grepping ":PASS" misses pio's "[PASSED]", grepping +# "[FAILED]" misses Unity's ":FAIL:"). So every regex below matches BOTH spellings: +# pass: Unity per-assertion ":PASS" | pio per-suite "[PASSED]" | summary "N succeeded" +# fail: Unity per-assertion ":FAIL:" | pio per-suite "[FAILED]" | summary "M failed" +# error: pio build/crash "[ERRORED]" | Unity "M Failures" | compiler "error:" +# Match \b after :PASS/:FAIL so ":PASSED"/":FAILED" forms are also caught either way. +FAIL_RE=':FAIL\b|\[FAILED\]|\[ERRORED\]|[1-9][0-9]* failed|[0-9]+ Tests [1-9][0-9]* Failures|error:|undefined reference|Segmentation fault|terminate called|SIGHUP|SIGSEGV|SIGABRT' +# Positive proof tests actually ran & passed (absence != success). Accept any pass spelling: +# the per-test/per-suite tokens OR a success summary line. +PASS_RE=':PASS\b|\[PASSED\]|test cases: *[0-9]+ succeeded|[0-9]+ Tests 0 Failures' +# Sanitizer (ASan/LSan/UBSan/TSan) fault signatures. The coverage build is sanitizer-instrumented +# and aborts NON-ZERO at exit on a fault — most often a LeakSanitizer leak — AFTER every test has +# already printed [PASSED]. pio then reports [ERRORED]/SIGHUP with no :FAIL: anywhere, so it +# masquerades as a phantom "N-1 of N succeeded". See .notes/test-passfail-filter.md. +# Match only real FAULT lines, never the benign "AddressSanitizer: failed to intercept '...'" +# startup noise that prints on every sanitizer run (it'd mislabel a normal [FAILED] as a leak). +# Formats per LLVM/Google sanitizer docs: ASan/LSan emit "==PID==ERROR: : ...", UBSan emits +# "file:line:col: runtime error: ...", TSan emits "WARNING: ThreadSanitizer: ..."; all close with +# a "SUMMARY: : ..." line (LSan-under-ASan reports its SUMMARY as "AddressSanitizer"). +SAN_RE='(ERROR|WARNING): (Address|Leak|Thread|UndefinedBehavior)Sanitizer:|SUMMARY: (Address|Leak|Thread|UndefinedBehavior)Sanitizer:|Direct leak of|Indirect leak of|detected memory leaks|heap-use-after-free|heap-buffer-overflow|stack-buffer-overflow|attempting double-free|LeakSanitizer has encountered a fatal error|runtime error:' + +# Suites that produced a per-suite verdict. pio emits "coverage:test_x [PASSED|FAILED|ERRORED]"; +# a SKIPPED suite (hardware-only on native) is "accounted for" too, so it doesn't read as missing. +mapfile -t RAN_SUITES < <(grep -oE "${ENV}:test_[a-z0-9_]+ \[(PASSED|FAILED|ERRORED)\]" "$LOG" | + sed -E "s/^${ENV}:(test_[a-z0-9_]+) .*/\1/" | sort -u) +RAN_COUNT=${#RAN_SUITES[@]} +# Suites pio explicitly skipped (don't count these as "missing" in the canonical cross-check). +mapfile -t SKIPPED_SUITES < <(grep -oE "${ENV}:test_[a-z0-9_]+.*\bSKIPPED\b" "$LOG" | + grep -oE "test_[a-z0-9_]+" | sort -u) + +verdict_red() { + local detail bin + detail="$(grep -nE '\[FAILED\]|:FAIL:|\[ERRORED\]' "$LOG" | head -3 | sed 's/^/ /')" + echo "" + echo "RED — failures detected:" + [[ -n $detail ]] && echo "$detail" + grep -E 'test cases:' "$LOG" | tail -1 | sed 's/^/ /' + + # Path to the test binary for the "run it bare" hint. For native/coverage the test program is + # the env executable (e.g. .pio/build/coverage/meshtasticd), NOT a file named 'program'. + bin="$(find ".pio/build/${ENV}" -maxdepth 1 -type f -executable ! -name '*.so' 2>/dev/null | head -1)" + [[ -z $bin ]] && bin=".pio/build/${ENV}/ (build it first: $PIO test -e ${ENV} ${FILTER:+-f $FILTER} --without-testing)" + + # Sanitizer fault (ASan/LSan/UBSan/TSan): name the real cause instead of "build/crash error". + if grep -qE "$SAN_RE" "$LOG"; then + grep -nE "$SAN_RE" "$LOG" | head -4 | sed 's/^/ /' + echo " -> sanitizer fault: if every test above is PASS, this is an exit-time abort, not a failed assertion." + echo " -> read the full report by running the binary BARE (gdb hides it via ptrace): ./$bin 2>&1 | tail -40" + echo "RESULT: RED sanitizer fault — $(grep -ohE 'SUMMARY: [A-Za-z]+Sanitizer:.*' "$LOG" | tail -1 || echo 'see report above')" + exit 1 + fi + + # All tests passed but the process still aborted at EXIT (ERRORED/SIGHUP/SIGABRT) and the + # sanitizer report was swallowed by the runner (often surfaced only as SIGHUP). Almost always a + # sanitizer fault — point at how to surface it rather than calling it a generic crash. + if grep -qE "$PASS_RE" "$LOG" && grep -qE '\[ERRORED\]|SIGHUP|SIGABRT' "$LOG" && ! grep -qE ':FAIL\b|\[FAILED\]' "$LOG"; then + echo " -> all tests passed but the process aborted at EXIT — likely an ASan/LSan fault whose report" + echo " the runner swallowed (commonly shown as SIGHUP). Run the binary BARE to see it: ./$bin 2>&1 | tail -40" + echo "RESULT: RED exit-time abort (tests passed; likely sanitizer — see hint above)" + exit 1 + fi + + echo "RESULT: RED $(grep -oE '[0-9]+ failed' "$LOG" | tail -1 || echo 'build/crash error')" + exit 1 +} + +# RED: pio non-zero, any failure marker, or no positive summary at all (build died early). +if [[ $PIO_RC -ne 0 ]] || grep -qE "$FAIL_RE" "$LOG"; then + verdict_red +fi +if ! grep -qE "$PASS_RE" "$LOG"; then + echo "" + echo "RESULT: RED no success summary found (build error / no tests ran?) — see log" + exit 1 +fi + +# Canonical-count rating suffix — appended to every verdict line so the result is always +# rated against the registered total, not just the directory count. +# If the two counts diverge (suite added/removed without updating native-suite-count), that +# is itself surfaced as AMBER before we reach any verdict. +canonical_rating() { + if [[ -n $CANONICAL_COUNT ]]; then + echo "[canonical: ${RAN_COUNT}/${CANONICAL_COUNT}]" + fi +} + +# AMBER: directory count disagrees with native-suite-count — file needs updating. +if [[ -n $CANONICAL_COUNT && $EXPECTED_COUNT -ne $CANONICAL_COUNT ]]; then + echo "" + if [[ $EXPECTED_COUNT -gt $CANONICAL_COUNT ]]; then + echo "RESULT: AMBER test/ has $EXPECTED_COUNT suite directories but native-suite-count says $CANONICAL_COUNT — update test/native-suite-count after registering new suites" + else + echo "RESULT: AMBER test/ has $EXPECTED_COUNT suite directories but native-suite-count says $CANONICAL_COUNT — update test/native-suite-count after removing suites" + fi + exit 2 +fi + +# AMBER: individual test cases were skipped (Unity TEST_IGNORE → :IGNORE: in output). +# Applies to both full and filtered runs — a skipped test case is a lost signal either way. +mapfile -t IGNORED_TESTS < <(grep -oE '[^:]+:[0-9]+:[^:]+:IGNORE:.*' "$LOG" 2>/dev/null | sed 's/:IGNORE:.*//' | sort -u) +IGNORED_COUNT=${#IGNORED_TESTS[@]} +if [[ $IGNORED_COUNT -gt 0 ]]; then + IGNORE_DETAIL="$(printf '%s\n' "${IGNORED_TESTS[@]}" | head -5 | sed 's/^/ /')" + echo "" + echo "$IGNORE_DETAIL" + echo "" + echo "RESULT: AMBER ${IGNORED_COUNT} test case(s) ignored $(canonical_rating)" + exit 2 +fi + +# AMBER: full run only — a canonical suite neither ran NOR was explicitly skipped (silently missing). +ACCOUNTED_COUNT=$((RAN_COUNT + ${#SKIPPED_SUITES[@]})) +if [[ -z $FILTER && $ACCOUNTED_COUNT -lt $EXPECTED_COUNT ]]; then + missing=() + for s in "${ALL_SUITES[@]}"; do + printf '%s\n' "${RAN_SUITES[@]}" "${SKIPPED_SUITES[@]}" | grep -qx "$s" || missing+=("$s") + done + echo "" + echo "RESULT: AMBER ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (missing: ${missing[*]}) — all that ran passed $(canonical_rating)" + exit 2 +fi + +# FILTERED: a -f run completed cleanly. Suites outside the filter were intentionally not run; +# this is not a quality signal and is distinct from suites that went missing unexpectedly. +if [[ -n $FILTER ]]; then + not_run=() + for s in "${ALL_SUITES[@]}"; do + printf '%s\n' "${RAN_SUITES[@]}" "${SKIPPED_SUITES[@]}" | grep -qx "$s" || not_run+=("$s") + done + echo "RESULT: FILTERED ${RAN_COUNT}/${EXPECTED_COUNT} suites ran (not run: ${not_run[*]}) — filtered: $FILTER $(canonical_rating)" + exit 3 +fi + +# GREEN: all canonical suites ran, all passed, no ignored test cases. +echo "RESULT: GREEN ${RAN_COUNT}/${EXPECTED_COUNT} suites passed $(canonical_rating)" +exit 0 diff --git a/bin/seed-json-to-proto.py b/bin/seed-json-to-proto.py new file mode 100755 index 00000000000..80eb34d63b5 --- /dev/null +++ b/bin/seed-json-to-proto.py @@ -0,0 +1,342 @@ +#!/usr/bin/env python3 +"""Compile a committed seed JSONL into a binary meshtastic_NodeDatabase v25 proto. + +The input is produced by `bin/gen-fake-nodedb-seed.py`. Timestamps in the JSONL +are stored as `*_offset_sec` (seconds before "now"); this script resolves them +to absolute epochs using `--now-epoch` (default: current wall clock). + +Output is a raw `pb_encode`-compatible binary that can be dropped at +`/prefs/nodes.proto` on the device (Portduino prefs dir or hardware via +XModem) and loaded by `NodeDB::loadFromDisk` at boot. + +Wire format reference: + protobufs/meshtastic/deviceonly.proto (NodeDatabase, NodeInfoLite, sat entries) + src/mesh/NodeDB.h:467-484 (bitfield bit positions) + src/mesh/NodeDB.cpp:1523-1524 (pb_decode entry point) +""" + +from __future__ import annotations + +import argparse +import json +import pathlib +import sys +import time +from typing import Any + +# Prefer the in-tree generated Python protobuf bindings (bin/_generated/meshtastic_v25/) +# because the firmware branch's protos (v25 NodeDatabase satellite arrays, slim +# NodeInfoLite) are typically newer than what the PyPI `meshtastic` package +# ships. Run `bin/regen-py-protos.sh` to (re)generate. +# +# Namespace note: the local bindings live under `meshtastic_v25` (NOT `meshtastic`) +# to avoid shadowing the PyPI `meshtastic` package — bin/regen-py-protos.sh +# post-processes the protoc output to rename the package. +_HERE = pathlib.Path(__file__).resolve().parent +_LOCAL_PROTO_DIR = _HERE / "_generated" +if _LOCAL_PROTO_DIR.is_dir(): + sys.path.insert(0, str(_LOCAL_PROTO_DIR)) + +try: + from meshtastic_v25.deviceonly_pb2 import ( # type: ignore[import-not-found] + NodeDatabase, + NodeInfoLite, + NodePositionEntry, + NodeTelemetryEntry, + NodeEnvironmentEntry, + NodeStatusEntry, + PositionLite, + ) + from meshtastic_v25.mesh_pb2 import HardwareModel, Position, StatusMessage # type: ignore[import-not-found] + from meshtastic_v25.config_pb2 import Config # type: ignore[import-not-found] + from meshtastic_v25.telemetry_pb2 import DeviceMetrics, EnvironmentMetrics # type: ignore[import-not-found] +except ImportError as local_err: + # Fall back to the PyPI package if in-tree bindings haven't been generated. + # Will fail the v25 assertion below if the PyPI package predates the + # satellite-DB schema, but at least gives a clear "run regen-py-protos.sh" + # error message instead of an opaque ImportError. + try: + from meshtastic.protobuf.deviceonly_pb2 import ( + NodeDatabase, + NodeInfoLite, + NodePositionEntry, + NodeTelemetryEntry, + NodeEnvironmentEntry, + NodeStatusEntry, + PositionLite, + ) + from meshtastic.protobuf.mesh_pb2 import HardwareModel, Position, StatusMessage + from meshtastic.protobuf.config_pb2 import Config + from meshtastic.protobuf.telemetry_pb2 import DeviceMetrics, EnvironmentMetrics + except ImportError as pypi_err: + print( + "ERROR: could not import meshtastic protobuf bindings.\n" + " In-tree generation: run `bin/regen-py-protos.sh` (requires protoc).\n" + " PyPI fallback: `pip install meshtastic` (may lag firmware branch).\n" + f" local error (meshtastic_v25): {local_err}\n" + f" pypi error (meshtastic.protobuf): {pypi_err}", + file=sys.stderr, + ) + sys.exit(1) + +# Fail loudly if bindings predate v25 (no satellite arrays). +assert ( + hasattr(NodeDatabase, "DESCRIPTOR") + and "positions" in NodeDatabase.DESCRIPTOR.fields_by_name +), ( + "Loaded meshtastic bindings are older than v25 (NodeDatabase.positions missing). " + "Run `bin/regen-py-protos.sh` against the in-tree protobufs/ submodule." +) + +# --------------------------------------------------------------------------- +# Bitfield bit positions (mirror src/mesh/NodeDB.h:467-484). +# --------------------------------------------------------------------------- +BIT_IS_KEY_MANUALLY_VERIFIED = 0 +BIT_IS_MUTED = 1 +BIT_VIA_MQTT = 2 +BIT_IS_FAVORITE = 3 +BIT_IS_IGNORED = 4 +BIT_HAS_USER = 5 +BIT_IS_LICENSED = 6 +BIT_IS_UNMESSAGABLE = 7 +BIT_HAS_IS_UNMESSAGABLE = 8 + +BITFIELD_LAYOUT = ( + # JSON key bit position + ("is_key_manually_verified", BIT_IS_KEY_MANUALLY_VERIFIED), + ("is_muted", BIT_IS_MUTED), + ("via_mqtt", BIT_VIA_MQTT), + ("is_favorite", BIT_IS_FAVORITE), + ("is_ignored", BIT_IS_IGNORED), + ("has_user", BIT_HAS_USER), + ("is_licensed", BIT_IS_LICENSED), + ("is_unmessagable", BIT_IS_UNMESSAGABLE), + ("has_is_unmessagable", BIT_HAS_IS_UNMESSAGABLE), +) + + +def _pack_bitfield(bf: dict[str, bool]) -> int: + out = 0 + for key, shift in BITFIELD_LAYOUT: + if bf.get(key, False): + out |= (1 << shift) + return out + + +def _validate_node(node: dict[str, Any]) -> None: + """Friendly errors so hand-editors get clear feedback.""" + if "num" not in node or not isinstance(node["num"], str): + raise ValueError(f"node missing/invalid 'num' (must be hex string): {node!r}") + if "long_name" not in node: + raise ValueError(f"node {node['num']}: missing 'long_name'") + if len(node["long_name"]) > 24: + raise ValueError( + f"node {node['num']}: long_name {node['long_name']!r} is " + f"{len(node['long_name'])} chars; max 24 (nanopb max_size:25 minus NUL)" + ) + if "short_name" in node: + # short_name max_size:5 (incl. NUL) → 4 bytes of content. + # Char count is irrelevant — emojis with variation selectors (e.g. ❄️ = 6 B) + # would slip past a `len(str) > 4` check. Always measure bytes. + b = node["short_name"].encode("utf-8") + if len(b) > 4: + raise ValueError( + f"node {node['num']}: short_name {node['short_name']!r} is " + f"{len(b)} bytes UTF-8; max 4 (nanopb max_size:5 minus NUL)" + ) + pk = node.get("public_key_hex", "") + if pk and len(pk) != 64: + raise ValueError( + f"node {node['num']}: public_key_hex must be 64 hex chars or empty; " + f"got {len(pk)} chars" + ) + if pk: + try: + bytes.fromhex(pk) + except ValueError as e: + raise ValueError(f"node {node['num']}: public_key_hex is not valid hex: {e}") + + +def _resolve_time( + node: dict[str, Any], + field_absolute: str, + field_offset: str, + now_epoch: int, +) -> int: + """If `field_absolute` is set, use it; else compute `now_epoch - offset`.""" + if field_absolute in node and node[field_absolute] is not None: + return int(node[field_absolute]) + offset = node.get(field_offset, 0) + return max(0, int(now_epoch) - int(offset)) + + +def _build_node_info_lite(node: dict[str, Any], now_epoch: int) -> NodeInfoLite: + _validate_node(node) + info = NodeInfoLite() + info.num = int(node["num"], 16) if isinstance(node["num"], str) else int(node["num"]) + info.long_name = node.get("long_name", "") + info.short_name = node.get("short_name", "") + # Enum lookups will raise ValueError on unknown names — that's exactly what we want. + info.hw_model = HardwareModel.Value(node.get("hw_model", "UNSET")) + info.role = Config.DeviceConfig.Role.Value(node.get("role", "CLIENT")) + pk_hex = node.get("public_key_hex", "") + if pk_hex: + info.public_key = bytes.fromhex(pk_hex) + info.snr = float(node.get("snr", 0.0)) + info.channel = int(node.get("channel", 0)) + if "hops_away" in node: + # `optional uint32 hops_away = 9;` — in Python protobuf, assigning the + # field implicitly sets HasField("hops_away") to True. No has_hops_away + # setter exists (unlike the C++ nanopb-generated header). + info.hops_away = int(node["hops_away"]) + info.next_hop = int(node.get("next_hop", 0)) + info.last_heard = _resolve_time(node, "last_heard", "last_heard_offset_sec", now_epoch) + info.bitfield = _pack_bitfield(node.get("bitfield", {})) + return info + + +def _build_position_entry(num: int, pos: dict[str, Any], now_epoch: int) -> NodePositionEntry: + entry = NodePositionEntry() + entry.num = num + pl = PositionLite() + # Firmware stores lat/long as int32 in 1e-7 degrees. + pl.latitude_i = int(round(float(pos["latitude"]) * 1e7)) + pl.longitude_i = int(round(float(pos["longitude"]) * 1e7)) + pl.altitude = int(pos.get("altitude", 0)) + pl.time = _resolve_time(pos, "time", "time_offset_sec", now_epoch) + pl.location_source = Position.LocSource.Value(pos.get("location_source", "LOC_UNSET")) + entry.position.CopyFrom(pl) + return entry + + +def _build_telemetry_entry(num: int, tel: dict[str, Any]) -> NodeTelemetryEntry: + entry = NodeTelemetryEntry() + entry.num = num + dm = DeviceMetrics() + if "battery_level" in tel: + dm.battery_level = int(tel["battery_level"]) + if "voltage" in tel: + dm.voltage = float(tel["voltage"]) + if "channel_utilization" in tel: + dm.channel_utilization = float(tel["channel_utilization"]) + if "air_util_tx" in tel: + dm.air_util_tx = float(tel["air_util_tx"]) + if "uptime_seconds" in tel: + dm.uptime_seconds = int(tel["uptime_seconds"]) + entry.device_metrics.CopyFrom(dm) + return entry + + +def _build_environment_entry(num: int, env: dict[str, Any]) -> NodeEnvironmentEntry: + entry = NodeEnvironmentEntry() + entry.num = num + em = EnvironmentMetrics() + if "temperature" in env: + em.temperature = float(env["temperature"]) + if "relative_humidity" in env: + em.relative_humidity = float(env["relative_humidity"]) + if "barometric_pressure" in env: + em.barometric_pressure = float(env["barometric_pressure"]) + if "iaq" in env: + em.iaq = int(env["iaq"]) + entry.environment_metrics.CopyFrom(em) + return entry + + +def _build_status_entry(num: int, status: dict[str, Any]) -> NodeStatusEntry: + # `StatusMessage` (mesh.proto:1445) has a single `string status` field. + entry = NodeStatusEntry() + entry.num = num + sm = StatusMessage() + if "status" in status: + sm.status = str(status["status"]) + entry.status.CopyFrom(sm) + return entry + + +def compile_jsonl_to_proto(jsonl_path: pathlib.Path, now_epoch: int) -> bytes: + """Read a seed JSONL and return the encoded NodeDatabase bytes.""" + lines = jsonl_path.read_text(encoding="utf-8").splitlines() + if not lines: + raise ValueError(f"{jsonl_path} is empty") + meta_line = lines[0] + meta_obj = json.loads(meta_line) + meta = meta_obj.get("_meta", {}) + version = meta.get("version") + if version != 25: + raise ValueError( + f"{jsonl_path}: meta version is {version!r}; this compiler " + f"requires version=25. Regenerate the seed with the matching tooling." + ) + + db = NodeDatabase() + db.version = 25 + + for ln, raw in enumerate(lines[1:], start=2): + raw = raw.strip() + if not raw: + continue + try: + node = json.loads(raw) + except json.JSONDecodeError as e: + raise ValueError(f"{jsonl_path}:{ln} JSON parse error: {e}") + + num = int(node["num"], 16) if isinstance(node["num"], str) else int(node["num"]) + + # Header + info = _build_node_info_lite(node, now_epoch) + db.nodes.append(info) + + # Satellites (nullable) + if node.get("position"): + db.positions.append(_build_position_entry(num, node["position"], now_epoch)) + if node.get("telemetry"): + db.telemetry.append(_build_telemetry_entry(num, node["telemetry"])) + if node.get("environment"): + db.environment.append(_build_environment_entry(num, node["environment"])) + if node.get("status"): + db.status.append(_build_status_entry(num, node["status"])) + + return db.SerializeToString() + + +def main(argv: list[str]) -> int: + p = argparse.ArgumentParser( + description="Compile a seed JSONL into a binary v25 NodeDatabase proto.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + p.add_argument("--in", dest="in_path", required=True, help="Input seed JSONL.") + p.add_argument("--out", required=True, help="Output binary .proto path.") + p.add_argument( + "--now-epoch", + type=int, + default=None, + help="Pin 'now' to this Unix epoch (for byte-identical CI). Default: time.time().", + ) + args = p.parse_args(argv) + + in_path = pathlib.Path(args.in_path) + if not in_path.is_file(): + print(f"input not found: {in_path}", file=sys.stderr) + return 2 + + now_epoch = args.now_epoch if args.now_epoch is not None else int(time.time()) + + try: + encoded = compile_jsonl_to_proto(in_path, now_epoch) + except ValueError as e: + print(f"ERROR: {e}", file=sys.stderr) + return 3 + + out_path = pathlib.Path(args.out) + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.write_bytes(encoded) + print( + f"compiled {in_path} -> {out_path} ({len(encoded)} bytes, now_epoch={now_epoch})", + file=sys.stderr, + ) + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/bin/shame.py b/bin/shame.py deleted file mode 100644 index f2253bfdcc5..00000000000 --- a/bin/shame.py +++ /dev/null @@ -1,95 +0,0 @@ -import sys -import os -import json -from github import Github - -def parseFile(path): - with open(path, "r") as f: - data = json.loads(f) - for file in data["files"]: - if file["name"].endswith(".bin"): - return file["name"], file["bytes"] - -if len(sys.argv) != 4: - print(f"expected usage: {sys.argv[0]} ") - sys.exit(1) - -pr_number = int(sys.argv[1]) - -token = os.getenv("GITHUB_TOKEN") -if not token: - raise EnvironmentError("GITHUB_TOKEN not found in environment.") - -repo_name = os.getenv("GITHUB_REPOSITORY") # "owner/repo" -if not repo_name: - raise EnvironmentError("GITHUB_REPOSITORY not found in environment.") - -oldFiles = sys.argv[2] -old = set(os.path.join(oldFiles, f) for f in os.listdir(oldFiles) if os.path.isfile(f)) -newFiles = sys.argv[3] -new = set(os.path.join(newFiles, f) for f in os.listdir(newFiles) if os.path.isfile(f)) - -startMarkdown = "# Target Size Changes\n\n" -markdown = "" - -newlyIntroduced = new - old -if len(newlyIntroduced) > 0: - markdown += "## Newly Introduced Targets\n\n" - # create a table - markdown += "| File | Size |\n" - markdown += "| ---- | ---- |\n" - for f in newlyIntroduced: - name, size = parseFile(f) - markdown += f"| `{name}` | {size}b |\n" - -# do not log removed targets -# PRs only run a small subset of builds, so removed targets are not meaningful -# since they are very likely to just be not ran in PR CI - -both = old & new -degradations = [] -improvements = [] -for f in both: - oldName, oldSize = parseFile(f) - _, newSize = parseFile(f) - if oldSize != newSize: - if newSize < oldSize: - improvements.append((oldName, oldSize, newSize)) - else: - degradations.append((oldName, oldSize, newSize)) - -if len(degradations) > 0: - markdown += "\n## Degradation\n\n" - # create a table - markdown += "| File | Difference | Old Size | New Size |\n" - markdown += "| ---- | ---------- | -------- | -------- |\n" - for oldName, oldSize, newSize in degradations: - markdown += f"| `{oldName}` | **{oldSize - newSize}b** | {oldSize}b | {newSize}b |\n" - -if len(improvements) > 0: - markdown += "\n## Improvement\n\n" - # create a table - markdown += "| File | Difference | Old Size | New Size |\n" - markdown += "| ---- | ---------- | -------- | -------- |\n" - for oldName, oldSize, newSize in improvements: - markdown += f"| `{oldName}` | **{oldSize - newSize}b** | {oldSize}b | {newSize}b |\n" - -if len(markdown) == 0: - markdown = "No changes in target sizes detected." - -g = Github(token) -repo = g.get_repo(repo_name) -pr = repo.get_pull(pr_number) - -existing_comment = None -for comment in pr.get_issue_comments(): - if comment.body.startswith(startMarkdown): - existing_comment = comment - break - -final_markdown = startMarkdown + markdown - -if existing_comment: - existing_comment.edit(body=final_markdown) -else: - pr.create_issue_comment(body=final_markdown) diff --git a/bin/size_report.py b/bin/size_report.py new file mode 100644 index 00000000000..5107a39e129 --- /dev/null +++ b/bin/size_report.py @@ -0,0 +1,383 @@ +#!/usr/bin/env python3 + +"""Compare firmware size reports and generate a markdown summary. + +Usage: + size_report.py [--baseline