feat: NixOS migration infrastructure#433
Open
mrosseel wants to merge 7 commits into
Open
Conversation
mrosseel
changed the title
upstream defines utils.tetra3_dir as the inner package path (python/PiFinder/tetra3/tetra3) and every sys.path.append(tetra3_dir) site relies on that. migration had the short submodule-root path plus an extra sys.path.append(tetra3_dir / 'tetra3') workaround in solver_main.py and ui/preview.py, but solver.py never got the workaround — so 'from tetra3 import cedar_detect_client' fails when the inner module does the bare 'import cedar_detect_pb2'. Take upstream's pattern verbatim: long tetra3_dir, single sys.path.append, no workaround. Fixes PR brickbots#433's nox ui_tests failure.
Two coupled changes for upstream's new test_all_ui_modules_covered guard (PR brickbots#438): - Wire UIMigrationConfirm and UIMigrationProgress into _DYNAMIC_IDS with item_definition fixtures. Their __init__ methods use .get() with defaults so a stub version_info dict exercises construction + key handlers. - UIReleaseNotes stays in _COVERAGE_SKIP — its active() fetches markdown over HTTP and needs a network mock. - UIMigrationProgress.update() was crashing under the smoke harness because sys_utils mock returned MagicMock for percent/status. Coerce percent to int and accept status only as str; on bad data keep the prior value. This also hardens against a corrupt /tmp/nixos_migration_progress JSON file at runtime.
Dead code dropped from the tree:
* python/PiFinder/sys_utils_nixos.py (~591 lines): never imported,
get_sys_utils() has no NixOS dispatch path. The NixOS-side system
utilities ship inside the migration tarball as
python/PiFinder/sys_utils.py on the nixos branch.
* python/pyproject.toml dbus/gi mypy ignores: only made sense for the
above; belong on the nixos branch.
* python/scripts/migration_calc.py (~509 lines): described an
unimplemented A/B-partition layout; no caller in the active flow
(nixos_migration.sh uses nixos_migration_calc.py instead).
Out-of-tree (moved to local notes):
* MIGRATION_BRANCH_STATE.md (107 lines): internal hand-off notes, not
user-facing docs.
* python/scripts/test_migration_loopdev.sh (~498 lines): offline test
harness that re-implemented an older design (tar.gz + magic-header
staging) rather than invoking nixos_migration_init.sh — it had
already drifted from the real flow (which is tar.zst + RAM-staged).
Useful as a future starting point for a real integration test but
misleading to ship in the repo.
Documentation:
* migration_gate.txt: add header comment explaining the killswitch
contract; update _fetch_migration_gate parser to skip "#" lines so
the file is self-documenting without breaking semantics.
…-fail)
Three small fixes from the PR review:
- ui/software.py UIMigrationProgress.update: replace the redundant
`except (AttributeError, Exception)` with a targeted `AttributeError`
guard around the sys_utils.get_migration_progress() lookup (only
needed when running against sys_utils_fake). The helper itself
swallows OS/JSON errors and returns {}, so wrapping everything in
`except Exception` was hiding real failures from the polling loop.
- ui/software.py get_release_version: add timeout=REQUEST_TIMEOUT to the
requests.get call (it previously had none and would hang the UI
thread if GitHub stalled). Widen the except to RequestException so
Timeout, ReadTimeout, etc. are all caught.
- sys_utils.start_nixos_migration: hard-fail with ValueError when neither
migration_sha256_url nor migration_sha256 produces a value. Previously
the helper logged a warning and returned "", which the migration
script then treated as "skip checksum verification". An in-place OS
replacement must not run without integrity verification.
The initramfs WiFi-migration step generated NetworkManager .nmconnection files by interpolating the SSID and PSK directly from wpa_supplicant.conf into a heredoc. SSIDs containing characters with semantic meaning in NM keyfile format (semicolon, brackets, equals, leading/trailing whitespace) or in the filesystem (slash, NUL, "..") could break the connection file, the file name, or both. Failure mode: WiFi config goes missing after migration -> headless device is unreachable until re-flashed. Fixes: - Encode SSID as semicolon-separated hex bytes (ssid=4d;79;...). This is NM keyfile's standard binary form and is safe for any byte content including non-ASCII and special chars. - Escape the id= and psk= values for NM keyfile format: backslashes doubled, semicolons backslashed. - Sanitize the filename to [A-Za-z0-9._-]; empty / "." / ".." after sanitization fall back to "wifi". - Use printf %s instead of echo when feeding the parser, so SSIDs starting with "-" or containing backslash escapes are not mangled by echo's flag interpretation. Verified end-to-end with a sample wpa_supplicant.conf containing spaces, slashes, and a semicolon in the PSK -- files generated cleanly with the expected escaping.
2 tasks
mrosseel
force-pushed
the
migration
branch
8 times, most recently
from
678f552 to
18c22fd
Compare
ui: render migration status from frame 0, expose underlying error, and allow back/exit on terminal pre-start failure so the user isn't trapped on the failure screen.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Lets an existing PiFinder install migrate in-place to a NixOS-based PiFinder build, driven from the existing Software-update screen.
What's included
Python
PiFinder/ui/software.py— migration-confirm screen + on-device progress flow alongside the existing software update path. Validates the connected display is a supported migration display before allowing migration.PiFinder/sys_utils.py—start_nixos_migration(version_info)plumbing: requires a SHA256, forwards display class + resolution to the script.tests/test_software.py— unit coverage for the migration UI flow.Scripts (under
python/scripts/)nixos_migration.sh— pre-migration host phase: validates env, downloads + verifies the tarball, assembles the initramfs (busybox + fs tools + SPI kernel modules), configures/boot/config.txtto boot it.nixos_migration_init.sh— initramfs entry point: saves WiFi + user backup to RAM (per-file sizing withpifinder.logtruncation allowance), DDs the image, expands the partition, restores data, reboots into NixOS. WiFi keyfile emission hex-encodes the SSID, escapes the PSK, and sanitizes the filename.nixos_migration_calc.py— pre-flight checks (RAM, free space, WiFi mode, supported display, stock SD layout) with a JSON output mode for the shell wrapper. Migration aborts if/is not on/dev/mmcblk0p2or if the SD has more than the expected two partitions (USB/NVMe boot, custom layouts).migration_progress.c+ pre-compiledmigration_progressbinary — OLED progress UI rendered directly from the initramfs (no Python). Picks the SPI controller and layout fromMIGRATION_DISPLAY_CLASS/MIGRATION_DISPLAY_RESOLUTIONenv vars passed through the initramfs.Other
migration_gate.json— server-side config fetched from thereleasebranch ofbrickbots/PiFinder. Two fields:nixos_for_everyone(the gate — the migration UI only appears when this istrue) andnixos_url(the tarball location). The.sha256URL is derived fromnixos_url.noxfile.py— narrows mypy's first run toPiFinder/(avoids the brokentetra3symlink in the tree).Test plan
nox -s lint type_hints smoke_tests unit_testspasses.Notes for review
migration_progressbinary is pre-compiled and committed (~80 KB) so the initramfs is self-contained — one-shot binary for the migration release.SUPPORTED_DISPLAYSis duplicated betweensoftware.py(early UI gate) andnixos_migration_calc.py(pre-flight gate). Belt-and-braces; both must be updated together if a new display ships.