Deploy pi from main within minutes: gated pull vs CI push

[etrobert-bot]

Goal

Pi converges on merged main within minutes instead of nightly. Promptness is the
requirement ("starts ≤1 min after CI green"); strict liveness (live ≤1 min) is
explicitly not required. France box deferred — this is pi-only.

Invariant: never deploy raw main

Every deploy gates on all-builds green. That guarantees both that the commit
builds and that the aarch64 closure is already in Cachix (tower's post-build
hook pushes synchronously during the CI build). This is an improvement over the
status quo, where the nightly autoUpgrade pulls raw main with no CI gating
at all.

CI cannot catch builds-fine-but-misbehaves-at-runtime changes; those deploy
minutes after merge. Accepted: rollback is manual (ssh pi +
nixos-rebuild switch --rollback), physical access as backstop.

Two candidate implementations

Build both as separate PRs and compare working code; the loser doesn't merge.

A. Gated pull

• CI job (after all-builds, main only, contents: write) fast-forwards a
  deploy ref: git push origin HEAD:deploy.
• Pi's system.autoUpgrade retargets from main to deploy, dates = "*:0/1"
  (or *:0/2), drop randomizedDelaySec.
• ExecCondition on the nixos-upgrade service compares git ls-remote of
  deploy against a last-deployed-rev state file (configurationRevision is
  unset in this flake, so nixos-version --configuration-revision can't be the
  comparison source). Idle ticks cost one git ls-remote.
• Latency: ~CI + 3–5 min (flake eval runs on the Pi 4 every real deploy).
• No new credentials anywhere.

B. CI push

• Deploy job (after all-builds, push to main only) on a tower runner:
  nixos-rebuild switch --flake .#pi --target-host <pi>. Eval happens on tower;
  pi only receives store paths and activates — no on-pi eval.
• SSH key stored as a GitHub environment secret with the environment
  restricted to main — branch/PR workflow runs cannot read it, so "can push a
  branch" does not become "root on pi".
• Key is from=-pinned to tower in pi's authorized_keys, so an exfiltrated
  key is useless off tower.
• Existing nightly autoUpgrade kept unchanged as offline-catch-up backstop.
• Latency: ~CI + ≤1 min.

Comparison criteria

• End-to-end latency (merge → live on pi)
• Moving parts / custom code each design actually required
• Credential surface (A: none; B: main-scoped, from=-pinned key — verify the
  scoping holds in practice)
• Failure behavior when pi is offline or the deploy fails mid-way

Explicitly rejected

• Polling raw main faster: races CI/Cachix and wastes on-pi eval; frequency
  was never the bottleneck.
• comin: no use for its testing-branch feature over ssh pi +
  nixos-rebuild test; dependency not earning its place.
• Auto-rollback / health checks on pi: LAN physical access suffices (reconsider
  for the France box).
• Deploy windows and allowReboot: complexity with no observed problem behind
  it. Kernel updates still apply only on manual reboot.

Doc follow-through (whichever lands)

• Update "Pi auto-upgrades from main nightly — test before merging" in
  CLAUDE.md — merges now go live in minutes, so the discipline tightens.
• Same note in modules/lan-dns.nix if repeated there.

🤖 Generated with Claude Code

[etrobert-bot]

Both candidate implementations are built, reviewed, and green in CI:

• A (gated pull): feat(pi): deploy from CI-gated ref within a minute (candidate A: gated pull) #483 — CI fast-forwards a deploy ref after all-builds; pi polls it every minute, gated by a git ls-remote ExecCondition against a last-deployed-rev state file.
• B (CI push): feat(pi): deploy on CI green instead of waiting for nightly #482 — tower runner runs nixos-rebuild switch --target-host root@pi after all-builds; main-scoped pi-deploy environment secret, from=-pinned key; nightly autoUpgrade kept as backstop.

Comparison against the issue's criteria

Criterion	A (gated pull)	B (CI push)
Latency (merge → live)	CI + ~3–5 min (flake eval on the Pi 4 each real deploy; ≤1 min to start)	CI + ≤1 min (eval on tower; pi only activates)
Moving parts	1 CI job + pi-deploy-gate script (check/record state machine)	1 CI job; no custom script beyond the workflow step
Credential surface	None	Main-scoped environment secret, from=-pinned to tower — reviewer verified both scopings hold in practice
Pi offline	Ticks don't run; converges by itself when back	Deploy job fails red; nightly autoUpgrade catches up
Manual setup before live	None	Create pi-deploy environment (branch-restricted to main) + PI_DEPLOY_SSH_KEY secret; steps in DEPLOY-SETUP.md in the B worktree on tower

Both passed review round 1 with no blocking findings. Pick the winner, merge it (squash), and close the loser unmerged.

🤖 Generated with Claude Code

[etrobert-bot]

Candidate C (comin-based gated pull): #484 is now also built, reviewed (round 1, no blocking findings), and green in CI. Three-way comparison:

Criterion	A: gated pull (#483)	B: CI push (#482)	C: comin (#484)
Latency (merge → live)	CI + ~3–5 min (on-pi eval)	CI + ≤1 min (eval on tower)	Same as A
Custom code	pi-deploy-gate check/record state machine	None beyond the workflow step	None — comin's built-in fetcher/deployer
Dependencies	None	None	New third-party flake input (comin 0.13.0)
Credential surface	None	Main-scoped env secret, from=-pinned root key	None
Pi offline	Converges alone when back	Deploy job fails red; nightly catches up	Converges alone when back
Manual setup	None	GitHub environment + secret	None
Invariant coverage	Full (sole mechanism is gated)	Fast path only (nightly still pulls raw main)	Full (autoUpgrade removed, comin sole mechanism)

C-specific notes from review: before the first deploy ref exists comin just logs and keeps polling (verified in source); no redeploy loops; state survives reboots; its Prometheus exporter (port 4243) is unreachable off-loopback given pi's firewall. A and C share the identical CI deploy-ref job — the choice between them is "40-line bespoke gate script, zero deps" vs "zero bespoke code, one new dependency".

One of the three merges (squash); the other two close unmerged.

🤖 Generated with Claude Code