hs-uploader cannot auto-generate its SFTP key: /etc/hs-uploader/keys is read-only to the service

[rrobinett]

Found on the VM101 dasi-greenfield bring-up (2026-07-25, AC0G/T1).

Symptom: every wsprdaemon-tar ship logs

Saving key "/etc/hs-uploader/keys/id_ed25519_host" failed: Read-only file system
station ssh key auto-generation failed ... continuing without a key
WsprdaemonTarSftp: gw1/gw2 failed: Permission denied (publickey,password)
... attempting FTP fallback

so a greenfield host permanently limps on the FTP fallback. Worse, the fallback's client_upload_info.txt is built without a pubkey (generation failed), so the gateway can never auto-provision SFTP — the fallback loop is stable, not self-healing.

Cause: hs-uploader.service runs with systemd filesystem hardening (ProtectSystem) that leaves /etc/hs-uploader/keys read-only to the hsupload user, but the transport tries to ssh-keygen into that directory at first use.

Workaround applied on VM101: generated the key as root and chowned to hsupload:

ssh-keygen -q -t ed25519 -f /etc/hs-uploader/keys/id_ed25519_host -N '' -C 'hs-uploader@AC0G/T1'
chown -R hsupload:hsupload /etc/hs-uploader/keys

Proper fix options:

1. Generate the key at manifest-write time (smd admin uploader manifest --write runs as root) — preferred, keys exist before first ship; or
2. add ReadWritePaths=/etc/hs-uploader/keys to the unit; or
3. move the key path under /var/lib/hs-uploader/ (already writable StateDirectory).

B4 production never hit this because its key predates the hardening/cutover.

[rrobinett]

Two additions from the 2026-07-25 full greenfield re-run (VM destroyed and recreated from a stock Debian 13 cloud image):

1. smd admin validate says the fix is "the hs-uploader key self-generates on first upload; force it with `smd admin secrets`" — but self-generation is exactly what the read-only key dir breaks, and smd bringup never runs the secrets step. So a greenfield host ends the bringup with validate telling the operator the key will appear on its own, and it never does. Bringup should run the smd admin secrets generation (as root) as part of the flow.

2. Filename mismatch worth checking: validate looks for /etc/hs-uploader/keys/id_ed25519 while the wsprdaemon transport generates/uses /etc/hs-uploader/keys/id_ed25519_host — one of the two paths is stale.