Push to soft-nix.cachix.org from aaron (Phase 2)

[etrobert-bot]

Plan: Push to soft-nix.cachix.org from aaron (Phase 2)

Status: deferred — start only after Phase 1
(#150) ships and we have data on how often the manual path
is insufficient.

Context

Phase 1 covers tower and leod via the upstream
services.cachix-watch-store NixOS module. aaron is split off because
Determinate Nix already owns the post-build-hook slot, and the obvious
"just set post-build-hook" approach has three independent problems:

1. Single-valued setting. Nix exposes one post-build-hook. Setting it via
   determinateNix.customSettings.post-build-hook replaces Determinate's
   default at /nix/var/determinate/post-build-hook.sh, which feeds JSON
   events into /nix/var/determinate/intake.pipe for determinate-nixd. We'd
   silently break any Determinate feature that depends on those events.
2. Open hang bug. DeterminateSystems/nix-installer#1500 reports
   Determinate's own post-build-hook hanging on macOS, unresolved as of last
   check. Adding more work through that pathway is risky.
3. Synchronous blocking. Nix's post-build-hook blocks the build loop.
   The official nix.dev guide explicitly warns this "will make Nix slow or
   unusable when the network connection is slow or unreliable."

Motivation

Heavy darwin closures (e.g. stdenv-darwin regressions like
compiler-rt-libc PR #147) get built locally during verification, then CI
rebuilds the same closure from scratch because soft-nix doesn't have it. If
aaron pushed locally, CI would download instead of compile.

Frequency, though, is low — this only matters when stdenv-darwin
invalidates. A few times per year, not per week.

Step 1: cachix-push-last shell script (do this first)

Add a small wrapper as a new wrapped package
(pkgs/cachix-push-last/default.nix) following the existing
writeShellApplication pattern (e.g. pkgs/gen-commit-msg).

Behavior:

• Resolves an out-path: $1 if provided, else ./result.
• Reads the cachix auth token from ~/.config/cachix/cachix.dhall (set via
  cachix authtoken once on aaron — see "Token storage" below).
• Calls cachix push soft-nix <out-path>.
• Prints what it pushed.

Add a workflow note to CLAUDE.md (project, not global) under
### macOS specific (aaron):

When a darwin build invalidates a large closure (typically a stdenv-darwin
change), run cachix-push-last after a successful local build so CI does
not have to rebuild it.

Token storage on darwin

Two options, in preference order:

1. cachix authtoken (user-level): run cachix authtoken <token> once on
   aaron. Cachix stores it under ~/.config/cachix/. No agenix involvement.
   Simpler, no daemon needs to read the secret. The token is user-scope and
   leaks only as far as the soft user's home.
2. agenix-darwin secret (secrets/cachix-aaron-token.age): only worth it
   if Step 2 (SSE subscriber) ships, since that runs as a launchd daemon and
   can't read user dotfiles.

Start with option 1.

Why this is enough for now

• Solves the actual triggering scenario (fix(aaron): workaround compiler-rt 18 darwin build via overlay #147): the user knew they were doing
  a heavy build; one command after nix build succeeds and CI is unblocked.
• Zero conflict with Determinate.
• No new daemon, no token written to disk for a system service, no module.
• Manual = inspectable. We see exactly what gets pushed.

Exit criteria — when to escalate to Step 2

Track instances where:

• We forgot to run cachix-push-last and CI subsequently rebuilt a large
  darwin closure.
• We built something on aaron that CI would have wanted, but it was a routine
  build (not a "heavy" one we'd think to push manually).

If either happens more than ~3 times in 6 months, do Step 2.

Step 2: Determinate SSE event subscriber (only if Step 1 proves insufficient)

The supported way to react to Determinate-managed builds without touching
post-build-hook is the streaming HTTP API on the Unix socket at
/nix/var/determinate/determinate-nixd.socket. Determinate's
"Subscribing to post-build events" guide documents this.

Architecture

1. launchd daemon running as root (or _nixbld-equivalent — confirm
   during impl). Started via launchd.daemons.cachix-push-subscriber in
   modules/hosts/aaron/configuration.nix (or a darwin module under
   modules/).
2. Subscriber script (Bash or small Go/Rust binary, wrapped via
   writeShellApplication):
   • Opens curl --unix-socket /nix/var/determinate/determinate-nixd.socket http://localhost/events
   • Parses each SSE message; matches v=1, c=BuiltPathResponseEventV1.
   • Extracts the output paths.
   • Calls cachix push soft-nix <paths> with the auth token from
     /run/agenix/cachix-aaron-token.
3. agenix secret cachix-aaron-token.age with mode = "0400" and an
   owner the daemon can read. Add aaron to its publicKeys in
   secrets/secrets.nix.

Open questions to resolve at impl time

• Does determinate-nixd.socket exist on every darwin install? Yes if
  Determinate is installed (it is on aaron). Confirm by
  ls -la /nix/var/determinate/ once before starting.
• Does the SSE endpoint emit events for substituted paths or only locally
  built ones? Per Determinate docs the events are built-path events, so
  substituted paths should not trigger them — same semantics as the standard
  post-build-hook. Verify by triggering a substitution and confirming no
  event.
• agenix-darwin secret readable by launchd daemon. agenix-darwin mounts
  secrets under /run/agenix.d/<gen>/. The daemon's User must match the
  secret's owner. Check the agenix.secrets.<name>.owner and .group
  options on darwin.
• Reconnection. The SSE connection can drop. The launchd KeepAlive
  policy should restart the daemon on exit; the subscriber should use
  curl --retry or wrap the connection in a
  while true; do curl ...; sleep 1; done loop.

Test plan

1. Build aaron with the subscriber enabled.
2. nix build nixpkgs#hello --rebuild and confirm a new entry appears on the
   cachix dashboard.
3. nix build nixpkgs#hello (no rebuild, substituted from cache) and confirm
   no new push event in the daemon log.
4. Kill the daemon mid-build; confirm launchd restarts it and queued events
   are not lost (or, if they are, document that — it's acceptable for this
   use case).

Out of scope (both steps)

• Migrating off Determinate Nix.
• Setting post-build-hook in nix.custom.conf directly. Don't.
• Caching pi builds (covered in Phase 1's out-of-scope; pi rebuilds from CI).

References

• Phase 1 PR: feat(cachix): auto-push tower/leod builds to soft-nix cache #150
• Determinate post-build events:
  https://docs.determinate.systems/guides/post-build-events/
• Determinate hang issue:
  nix builds hang on post-build-hook DeterminateSystems/nix-installer#1500
• Cachix push docs: https://docs.cachix.org/pushing
• Triggering PR: fix(aaron): workaround compiler-rt 18 darwin build via overlay #147