`pv update` incurs a 10-second launchd throttle after switching app binary

[munezaclovis]

Summary

Updating the PV application can add an avoidable ~10-second delay while restarting the daemon. During a local update from PV 0.2.0 to 0.2.2, the first launchd spawn of the new binary failed with a launch-constraint violation. launchd then throttled the service for exactly ten seconds before a retry succeeded.

This was not caused by Managed Resource downloads. The Managed Resource phase refreshed/checked the manifest, found every installed resource current, and completed in approximately one second.

Environment

• PV update: 0.2.0 → 0.2.2
• macOS: 26.5.1 (25F80)
• Darwin: 25.5.0

Observed timeline

22:30:19        PV 0.2.2 binary activated; daemon stopped
22:30:19.768    xpcproxy rejected the first daemon spawn
22:30:19.768    launchd deferred the next spawn by 10 seconds
22:30:21.337    launchd reported the service was still throttled
22:30:28.084    launchd reported the service was still throttled
22:30:29.871    daemon spawned successfully
22:30:29–30     Managed Resource update job completed with summary `current`

Relevant unified-log entries:

xpcproxy exited due to OS_REASON_CODESIGNING | Launch Constraint Violation,
error info: c[5]p[1]m[1]e[0], (Constraint not matched) ...
/Users/clovismuneza/.pv/bin/releases/0.2.2/pv

service spawn deferred by 10 seconds due to throttle

Successfully spawned pv[...] because inefficient

The update command took approximately 11 seconds overall.

Resource-update evidence

• The daemon update job completed with summary current.
• Installed Managed Resource release directories and binaries had no mtimes near the update.
• No new Managed Resource artifact was installed.
• A remote resource-manifest refresh/comparison still occurred, as expected.

Likely cause

This portion is strongly supported but not yet reproduced in an isolated test:

1. The LaunchAgent uses the stable ~/.pv/bin/pv symlink.
2. Self-update switches that symlink from the previous release binary to the new release binary.
3. normalize_launch_agent considers the plist current because its text/path did not change.
4. restart_daemon_without_reconciliation therefore runs only launchctl kickstart -k; it does not run bootout and bootstrap.
5. The old and new binaries have different ad-hoc signing identifiers and CDHashes.
6. macOS rejects the first spawn of the new symlink target against its launch constraint, then launchd's normal failure throttle delays the retry by ten seconds.

Confidence:

• High: the ten-second launchd throttle caused the delay.
• High: no Managed Resource binaries were downloaded.
• Medium-high: reusing the existing launchd registration after changing the symlink target caused the constraint mismatch.

Expected behavior

After activating a new PV release, the daemon should start successfully on its first launch attempt. pv update should not silently depend on launchd's throttled retry.

Proposed narrow solution

When the PV application binary changes, reload the owned LaunchAgent registration before starting the daemon:

1. launchctl bootout the existing PV service, tolerating the existing already-unloaded condition.
2. launchctl bootstrap the current PV LaunchAgent plist.
3. launchctl kickstart -k the newly registered service.
4. Keep the current daemon health wait and rollback behavior.

The plist does not need to be rewritten if its contents are already current; only the launchd registration needs refreshing after activation.

Acceptance criteria

• An app update reloads the LaunchAgent registration after switching the active binary.
• The updated daemon starts on the first attempt without a launch-constraint violation or ten-second throttle.
• The current rollback path still restores and starts the previous release if the updated daemon fails health checks.
• Integration coverage asserts the app-update restart sequence and preserves the existing current-app/no-update behavior.
• No changes are made to Managed Resource update/download behavior as part of this fix.

Notes

TLS/project reconciliation began later and was not responsible for the foreground update duration.

[pullfrog]

Assessment

This is actionable. In crates/cli/src/commands/update.rs, normalize_launch_agent returns NotRequired when the plist already points at the stable ~/.pv/bin/pv symlink, and restart_daemon_without_reconciliation then performs only kickstart -k after activating the new symlink target. That is the exact path implicated by the launch-constraint failure; stale plists already receive bootout/bootstrap/kickstart, and the current-app path correctly performs no daemon restart.

Plan

1. Update the self-update lifecycle contract in DESIGN.md to require refreshing the owned launchd registration whenever an app release is activated, including rollback activation, while preserving the no-restart behavior when the app version is current.
2. Adjust crates/cli/src/commands/update.rs so LaunchAgent preflight still validates ownership and rewrites stale plist contents, but returns the validated plist path and always runs tolerant bootout, bootstrap, then kickstart -k after a binary symlink switch. Reuse the same sequence after restoring the previous release so rollback does not retain registration constraints for the failed binary.
3. Update the integration coverage in crates/cli/tests/update.rs to assert the reload sequence for an update with an already-current plist, retain the stale-plist sequence, verify the update lock remains held through reload and health checking, and assert rollback reloads registration again after restoring the old target. Keep the existing current-app test asserting no launchd operations and no changes to Managed Resource continuation behavior.
4. Verify with cargo nextest run -p cli --test update --locked, then run cargo fmt --all --check and cargo clippy --workspace --all-targets --all-features --locked -- -D warnings. No platform API change should be necessary because the required launchctl operations and already-unloaded tolerance already exist.

Task list (3/3 completed)

• Read issue details and repository guidance
• Trace pv update and launchd binary-switch behavior
• Assess actionability and produce implementation plan

  ^{｜ Implement plan ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using GPT Sol ｜ 𝕏}

[munezaclovis]

Follow-up incident investigation (2026-07-28)

I investigated the live machine with PV 0.2.2. No repair or state mutation was performed.

Result

There are three different things being conflated:

1. The original pv update incurs a 10-second launchd throttle after switching app binary #300 launchd throttle is real, but it recovered after 10 seconds and is not still active. The LaunchAgent is currently loaded/running, its socket is healthy, launchctl print reports last exit code = 0, and the daemon has stayed up since the later restart on July 26. The 0.2.0 and 0.2.2 binaries also still show different ad-hoc identifiers/CDHashes, so the original launch-constraint analysis remains well supported.
2. The current ai.test outage is the low-port pf path, not DNS, the daemon socket, Gateway backend ports, or a persistent update lock.
3. The lock/status UX makes old or short-lived failures look current. The same update.lock is also the daemon's global reconciliation/mutation lock, and pv status treats any failed job among the last 100 as a current whole-system failure.

Current live evidence

• pv update --check now succeeds in about 0.5 seconds and reports everything current.
• No process has ~/.pv/run/update.lock open now. The file remaining on disk is expected; only the kernel lock matters.
• Daemon: running; LaunchAgent: current; socket: healthy.
• DNS: ai.test resolves to 127.0.0.1 and ::1; the daemon is listening on both UDP and TCP 127.0.0.1:35353.
• Gateway: listening on 127.0.0.1:48080 and 127.0.0.1:48443.
• Directly bypassing the low-port redirect and connecting ai.test:443 to 127.0.0.1:48443 reaches the Project and completes its redirect chain with HTTP 200.
• Both 127.0.0.1:80 and 127.0.0.1:443 refuse connections.
• The PV-owned system anchor file still exists and correctly maps 80 → 48080 and 443 → 48443, but /etc/pf.conf no longer contains the PV rdr-anchor / load anchor reference.
• pv ports:status therefore reports the system pf.conf reference as missing; pv doctor gives the focused repair pv ports:install.

The likely cause of that drift is the macOS update, with high confidence from the timeline:

• softwareupdate --history: macOS 26.5.2 installed July 24 at 12:47:58 EDT.
• PV daemon log: daemon stopped at exactly 12:47:58 and restarted at 12:56:43.
• /etc/pf.conf metadata changed at 12:54:47, inside that update/reboot interval, and its current contents are the stock Apple file with no PV block.
• Successful Gateway reconciliations exist before the OS update; the first low-port readiness failures occur afterward.

At investigation time the machine is macOS 26.5.2 (25F84), so the issue's 26.5.1 environment line should probably be corrected for the July 26 reproduction.

Why the lock looked stuck

The lock was not held for a day. Two cases appear in history:

• Jobs 10/11 failed at July 26 22:30:29 EDT, during the real app update while its foreground lock was correctly still held. Those are historical.
• A Project reconciliation started July 28 21:10:56 EDT and held the same OS lock until its Gateway readiness probe timed out 60 seconds later at 21:11:57. During that minute, pv update --check would report “PV update is already in progress,” even though the owner was reconciliation rather than app update.

That behavior follows the current code: the reconciliation queue acquires UpdateLock for mutating work (reconciliation.rs). The wording is therefore misleading for non-update owners.

The persistent appearance in pv status is separate: status loads the last 100 jobs, includes every failed one, and sets overall = failed whenever that list is non-empty (status.rs). Plain output omits the stored timestamps. With only 18 jobs in this database, the transient July 26 failures will keep looking current for a long time, even after repair.

Additional independent defect found

A July 28 21:10:44 reconciliation failed once with:

filesystem error at ~/.pv/pv.db-wal: No such file or directory

The next reconciliation started 12 seconds later, so this was transient. Code inspection strongly suggests a TOCTOU race rather than database corruption:

• Database::open secures the database/WAL/SHM files before and after configuration (database.rs).
• secure_database_files checks path.exists() and then separately chmods/inspects the path (fs.rs); SQLite may remove an ephemeral WAL between those operations.
• The Project config watcher opens a writable/migrating Database on every 100 ms poll (watcher.rs), increasing the opportunity for the race.

I found no existing issue for either the WAL error or post-macOS-update pf.conf drift.

Scope recommendation

Keep #300 narrow: re-register launchd after switching the app symlink target, including rollback. Today's outage does not weaken that diagnosis.

Track these separately:

• recovery/UX when a macOS update removes PV's pf.conf reference;
• lock wording plus status semantics for historical/resolved failed jobs;
• the pv.db-wal existence/permission race.

The immediate live repair indicated by PV is pv ports:install (foreground, requires macOS authentication). I deliberately did not run it as part of this read-only investigation. After restoring the redirect, a later successful reconciliation can refresh the Gateway observation, but the current “any failed job in the last 100” status rule will still need separate handling.

Secondary observation: the current anchor renders IPv4 redirects only while DNS advertises ::1. That is not the cause of this outage—IPv4 low ports are also broken—but it is worth covering in the ongoing listener/port work.

---

Addendum: OS-update attribution, repair validation, and issue boundaries (2026-07-28)

The macOS update attribution is now definitive

The earlier timeline inference can be strengthened with Apple's local package receipt and install log:

$ pkgutil --file-info /private/etc/pf.conf
pkgid: com.apple.files.data-template
pkg-version: 26.5.2
install-time: 1784912087  # 2026-07-24 12:54:47 EDT

com.apple.files.data-template 26.5.2 owns both private/etc/pf.conf and private/etc/pf.anchors. The corresponding stock file is /System/Library/Templates/Data/private/etc/pf.conf. Before repair, the live file had stock-template contents/mtime, no PV reference, and a ctime of exactly 12:54:47. The install log then records Previous System Version : 26.5.1 (25F80), Current System Version : 26.5.2 (25F84) at 12:55:34.

So macOS 26.5.2 replaced the pf.conf entry point during the update. It left PV's separate anchor file behind, but without the main-ruleset rdr-anchor and load anchor lines that file was orphaned and never evaluated.

The signature hunch is also real, but it is a different event

macOS launch/code-signing enforcement caused #300, but the July 24 OS update was not the direct trigger for the July 26 rejection:

• The LaunchAgent plist has no explicit SpawnConstraint, while launchctl print reports managed LWCR | has LWCR.
• Both PV binaries validate on disk, but they are ad-hoc/linker-signed and have distinct exact designated requirements:
  • 0.2.0: identifier pv-666ff83cc813cb86, CDHash 11ae2cbe8cbac8ad80ab79b9e48ab88a1d9495ae
  • 0.2.2: identifier pv-c9d856e79b57e802, CDHash 9439946ff0982c122603198f5a2fcc597e14595f
• Apple's documentation describes an LWCR as identity facts cached for a service by launchd, and launch constraints can match signing identifiers or Code Directory hashes: cached launchd LWCR, launch-constraint diagnosis, code-signing requirements.
• The OS update reboot successfully relaunched the still-active 0.2.0 daemon on July 24 at 12:56:43. The violation appeared only after PV switched the stable symlink to 0.2.2 on July 26 without refreshing the launchd registration.

Thus the evidence still supports #300's narrow fix: re-register the owned LaunchAgent after changing the symlink target. The OS update and PV app update happened close together, but they produced independent failures.

Why a browser request reaches PF

This is intentional in the current design:

ai.test -> DNS 127.0.0.1 -> HTTPS default port 443
        -> PF rdr on lo0 -> 127.0.0.1:48443
        -> unprivileged FrankenPHP Gateway -> Project worker

The Gateway runs as the user and therefore listens on high ports; PF makes normal portless https://*.test URLs reach it without running the daemon as root. The main pf.conf must both evaluate the PV rdr-anchor and load /etc/pf.anchors/com.prvious.pv; merely leaving the anchor file on disk does nothing.

This architecture is deliberate in DESIGN.md, but it carries platform risk: Apple explicitly says PF is not product API and warns that rules can clash with macOS services or other software (TN3165). The July 24 replacement is concrete evidence of that fragility and warrants a separate architecture/recovery discussion rather than changing #300.

Repair outcome

The user ran pv ports:install at 22:57 EDT. That restored both PV lines in /etc/pf.conf, reinstalled/reloaded the anchor, and immediately changed the live result:

• default and forced-IPv4 https://ai.test: HTTP 200 via 127.0.0.1:443;
• direct high-port bypass: HTTP 200 via 127.0.0.1:48443;
• forced IPv6: still connection-refused (secondary, not the original outage);
• no process currently owns the update lock;
• pv update --check --json succeeds and reports the app/resources current.

The repair also exposed additional independent diagnostics defects:

1. A normal-user pv status/doctor cannot inspect active PF state: /dev/pf is root-only, pfctl -s nat gets permission denied, and non-interactive sudo -n requires a password. Consequently a working repaired system is now reported as ports: unreadable.
2. pv ports:status reports the files as current but does not inspect loaded rules, so the two status surfaces disagree.
3. The old failed Gateway observation remains after ports:install, and every failed job among the last 100 still forces overall status to failed.
4. Gateway readiness chooses public 80/443 when the prepared anchor alone matches, even if the system reference is missing. That turns privileged PF drift into a 60-second timeout while holding the global mutation lock.

Recommended issue boundaries

Keep #300 solely about the launchd registration refresh. I recommend separate issues for:

1. macOS updates replacing PV's pf.conf reference and the supported recovery/architecture strategy;
2. Gateway readiness trusting prepared-only PF state and holding the global mutation lock for a doomed 60-second public-port probe;
3. normal-user PF diagnostics reporting a healthy active ruleset as unreadable, including the ports:status inconsistency;
4. status/doctor treating historical or superseded job failures and stale runtime observations as current health;
5. the generic mutation/reconciliation lock being surfaced as “PV update is already in progress”;
6. the independent pv.db-wal TOCTOU race.

The IPv6-only low-port gap should stay out of these incident issues and be filed separately only if it is not already covered by the intended IPv4-only degradation/warning design.

---

Follow-up issues created

The independent findings above are now tracked as:

• #304 — macOS updates replace PV's pf.conf reference and break .test routing
• #305 — Gateway readiness trusts prepared-only PF state and holds the mutation lock for 60 seconds
• #306 — pv status and pv doctor report healthy PF redirects as unreadable without sudo
• #307 — pv status keeps historical and superseded failures in current system health
• #308 — Global reconciliation lock is reported as “PV update is already in progress”
• #309 — Database permission hardening races with SQLite WAL removal

#300 remains scoped to the launchd registration refresh after switching the app binary.