fix(cachix-push): bound the push and alert on failure

[etrobert-bot]

Problem

Tower's post-build-hook (modules/cachix-push.nix) runs cachix push after each locally-built path. Because the hook blocks nix build and main CI never cancels (concurrency.cancel-in-progress: false), a stalled or misconfigured cache wedges every cache-miss build on main.

This bit us today: renaming a derivation forced local rebuilds, and cachix's push endpoint was degraded such that uploads stalled on an open connection — each stuck push ran ~15–20 min before completing, dragging out CI and delaying pi's deploy.

cachix's own retry logic (v1.11.1) bounds transient 5xx/429 failures (≤90 s cumulative backoff × 3 whole-op retries) but does not bound a stalled-open connection — there's no per-request response timeout — so a hung upload was effectively unbounded.

Fix

Wrap the push in timeout and make it non-fatal:

if ! timeout 180 cachix push soft-nix $OUT_PATHS; then
  ntfy publish --quiet \
    --title "cachix push failed on ${config.networking.hostName}" \
    "$OUT_PATHS"
fi

• timeout 180 — outer stop for the stalled-connection case cachix can't bound itself. A healthy push of a locally-built path takes <5 s, so this only trips when the cache is actually broken.
• Non-fatal — a push failing means the path just isn't cached, not that the build is broken; it must never fail/redden the build or wedge CI.
• Alert on any non-zero exit — keeps failures visible. Verified against the cachix source: a misconfig (401 bad/revoked token, 404 wrong cache name) throws a CachixException that is not retried, so it exits non-zero and fast — the ntfy ping fires on the first cache-miss build. Uses ntfy-wrapped via self.packages, mirroring modules/ntfy-failure-alerts.nix.

Trade-off

The hook fires per built path, so a sustained cache outage during a large rebuild can emit a burst of alerts (one per path). It self-limits and only happens when the cache is genuinely broken (zero noise when healthy). If that burst is annoying, a follow-up could move health-alerting to a periodic canary that pings once via the existing ntfy-failure@ infra.

Verification

• nix eval of nixosConfigurations.tower...toplevel.drvPath succeeds (module resolves self / ntfy-wrapped / coreutils).
• shellcheck passes on the rendered hook script.

🤖 Generated with Claude Code