fix(start-core): make install rollback crash-recoverable, and snapshot a stopped service

[helix-nine]

@dr-bonez — this is the rollback hardening from the Nextcloud data-loss investigation, with the ordering you asked for.

Why

The rollback is the only thing in the stack that deletes a populated volume during an update, and it was doing it non-atomically:

btrfs::delete_tree(&dst).await?;              // whole package volume root
crate::util::io::rename(&backup, &dst).await?;

Interrupt between those two and the service has neither copy. Worse, it doesn't look interrupted: handle_installed and Bind::pre_mount both recreate a missing data/<vol> as an empty dir, so the next boot sees a live root that exists, and recover_and_sweep's "backup with no live volume" test — the one recovery we had — doesn't fire. The stranded backup then gets deleted by the next attempt's snapshot_volumes_for_install, whose first act was "remove any stale backup". That last step is where the data actually goes.

What changed

Restore is a two-rename swap. live → <pkg>.restore-old, backup → live, drop the aside tree. Between the renames both copies exist, and .restore-old is a marker that survives the empty-skeleton recreation, so every interruption point maps to exactly one resolution:

on disk	resolution
restore-old + backup	live can only be a skeleton → replace it with the backup
restore-old, no backup	the rename landed → drop the aside tree
restore-old, no backup, no live	the aside tree is all that's left → rename it back
restore-old + backup + a live holding real files	refuse; don't pick a winner

resolve_pending_restore is called from ensure_volume_root — the funnel every path goes through before it trusts or creates the live root — plus the snapshot, the boot sweep, and remove_install_backup, which now refuses to discard a backup while a restore is in flight.

The snapshot stops destroying the only surviving copy. It snapshots to <pkg>.install-backup-tmp and swaps, so the previous rollback point is dropped only once its replacement exists.

A failed rollback is fatal and visible. It was .log_err()'d, and then the Ok path that followed called remove_install_backup — so a restore that died mid-swap got its rollback point deleted by the code meant to tidy up after success. It now notifies and propagates.

A failed first-time install over pre-existing data keeps that data. The Installing arm ran cleanup(soft=false) — an unconditional whole-root delete — before checking whether a backup existed. That arm is what a 0.3.x→0.4.0 package conversion runs under, since v0_3_6_alpha_0 resets packageData to {} and reinstalls.

The boot sweep is no longer readdir-order dependent. The orphan-reaping arm could delete a backup that an unvisited .restore-old marker still depended on. It now resolves every pending restore in a first pass. There's a test for exactly that.

ServiceRef::quiesce stops the main chain before the snapshot, leaving uninit to uninstall.

On the ordering, precisely

You asked for stop → snapshot → uninit. Snapshot-before-uninit was already true (service_map.rs:336 vs :376) and, being a CoW subvolume snapshot, uninit's writes can't reach it — so the genuinely new thing here is the quiesce, and I've kept the uninit ordering rather than claimed to have introduced it. What quiescing buys is not the snapshot's internal consistency (it's atomic, so a WAL store would replay fine) but: nothing is writing to the tree a later rollback will rename or delete, formats that aren't crash-safe on their own survive, and the rollback point is the state the user had when they pressed update.

quiesce() deliberately writes nothing to the status. Setting desired = Stopped would have left every successfully-updated service stopped (init() normalizes BackingUp/Restarting but passes Stopped through), and clearing started would let the actor's watch loop race a restart back in. It guards on is_initialized() because an uninitialized container's stop handler throws rather than no-oping, and it's bounded at 5 min — proceeding un-quiesced beats hanging an update on a wedged daemon.

Tests

11 new unit tests over the state machine, table-driven across {live present / skeleton / absent} × {backup} × {restore-old} × {owner installed / mid-flight / absent}. recover_and_sweep already took the volumes root as a parameter and the two protocol functions now do too, so these run on any filesystem with no btrfs, no root, and no fault injection — which matters, because the CI runner has neither CAP_SYS_ADMIN nor a btrfs workspace. cargo test -p start-core --lib: 556 passed.

Deliberately not in this PR

• The version bump. start-os/v0.4.0.1 is a cut tag, so I added a new ## [0.4.0.2] CHANGELOG heading, but I left the manifest half alone — root package.json, the Cargo.toml label + lock, version/v0_4_0_2.rs + version/mod.rs, and the docs release links. Cutting the next OS version is your call, and doing it here would collide with anyone else doing the same. Say the word and I'll add it as a second commit.
• cancel_install is an abandon, not a cancel. UnixRpcClient::request has no cancellation, so tokio::select! drops the future while the package's migration keeps running; teardown then falls to Drop for PersistentContainer, which spawns a detached destroy(None) that sends rpc::Exit with a real target — i.e. the package's uninit runs concurrently with the detached load(Undo) doing the renames. Two writers, one volume tree, no barrier. This is a plausible independent mechanism for the original incident and I think it wants its own change.
• Durability ordering. The Updating → Installed(old) DB flip still commits before the renames. The on-disk marker makes that recoverable, but a rollback: bool on UpdatingState (honoured at boot instead of always loading Retry) would also stop a crash mid-rollback from resuming the update the user cancelled.
• Dependency mounts can recreate a volume root while a restore is pending (Bind::pre_mount); snapshot_subvolume is non-recursive, so a nested subvolume would land in a backup as an empty dir.

Test plan

1. cargo test -p start-core --lib — 556 pass.
2. On a VM: update a running service, confirm it comes back running and healthy (the regression the status-writes discussion above is about).
3. Cancel an update mid-init; confirm the service returns to its old version with its data, and that <pkg>.install-backup is gone afterwards.
4. Simulate the incident: cancel, then kill -9 startd between the two renames (or drop a <pkg>.restore-old by hand), reboot, and confirm the boot sweep finishes the restore rather than starting the service on empty data.
5. Confirm a failed rollback surfaces as an error notification and leaves the package unloaded rather than silently "reverted".