Gateway readiness trusts prepared-only PF state and holds the mutation lock for 60 seconds

[munezaclovis]

Summary

Gateway readiness switches from the Gateway's high backend ports to public ports 80/443 when either active PF redirects match or merely the prepared anchor file matches. If macOS has removed PV's system pf.conf reference, the prepared file still matches even though no redirect is active.

Reconciliation then probes unreachable public ports for 60 seconds, marks a healthy Gateway process failed, and holds the global mutation lock for the duration.

This compounded the live incident described in #300's follow-up investigation.

Current behavior

The relevant selection is in gateway_readiness_ports:

if active_gateway_redirects_match_plan(plan)
    || prepared_gateway_redirects_match_plan(paths, plan)
{
    return GatewayReadinessPorts {
        http: PUBLIC_HTTP_PORT,
        https: PUBLIC_HTTPS_PORT,
    };
}

The reconciliation queue acquires the shared lock when mutating work is enqueued (reconciliation.rs) and releases it only when the queue becomes idle.

Observed sequence:

1. Prepared anchor correctly mapped 80 → 48080 and 443 → 48443.
2. The system anchor file also existed, but the system pf.conf reference was missing.
3. Gateway and Project worker processes were listening and direct high-port HTTPS returned 200.
4. Reconciliation selected ports 80/443 solely because the prepared file matched.
5. The public probe timed out after 60 seconds with connection refused.
6. During that minute, pv update --check reported that the update lock was held.

This also conflicts with the documented readiness example that the Gateway responds on its high HTTP/HTTPS ports (DESIGN.md).

Expected behavior

Prepared configuration alone must not be treated as proof that privileged redirects are loaded. Gateway process readiness and low-port system-integration health should remain distinguishable.

Acceptance criteria

• A current prepared anchor with a missing/stale system reference does not select a doomed public-port readiness probe.
• A live Gateway on its assigned high ports is not marked failed solely because privileged PF routing is stale.
• PF drift is recorded/reported as a system-integration repair requirement.
• Reconciliation does not hold the global mutation lock for the full public-port timeout in this state.
• Preserve safe handling for the macOS behavior noted in the code comment where direct target-port connections can hang while matching redirects are genuinely active.
• Add integration coverage for:
  • prepared current + system reference missing + backend healthy;
  • active redirects current;
  • backend genuinely unhealthy.
• Do not change the foreground privilege boundary for pv ports:install.

Related

• pv update incurs a 10-second launchd throttle after switching app binary #300
• Full incident evidence

[pullfrog]

Assessment

This issue is actionable. gateway_readiness_ports currently treats a current prepared anchor as equivalent to loaded PF redirects, so stale privileged integration can force a healthy Gateway through the 60-second public-port timeout. The queue lock behavior is not independently defective here: it spans a mutating reconciliation job by design, and fixing the doomed readiness path removes the minute-long lock hold. Existing pv status coverage already reports this exact PF drift as Ports: prepared-only; the implementation should preserve that repair signal and the foreground-only privilege boundary.

Plan

1. Refine Gateway readiness routing in crates/daemon/src/gateway.rs so prepared state is only desired configuration, not evidence of loaded redirects. Use public ports only when live redirects are confirmed to match. Use assigned high ports when PF is missing, inactive, mismatched, or the system anchor/reference is stale. For the important indeterminate case where system files are current but noninteractive live inspection is unavailable, use a bounded public/high-port fallback so genuinely active PF cannot strand a direct target-port probe.
2. Keep Gateway runtime health separate from PF integration health. A successful high-port probe records the Gateway as running even when low-port routing needs repair; retain the existing pv status/doctor prepared-only failure and pv ports:install repair guidance, without adding background privilege escalation or changing ports:install.
3. Add deterministic readiness-selection tests for confirmed active redirects, prepared-only/system-reference-missing state, confirmed inactive or mismatched redirects, and unreadable live PF with otherwise-current system files. Extend crates/daemon/tests/gateway_reconciliation.rs with full reconciliation coverage showing a prepared-only healthy backend becomes Running promptly, while a genuinely unhealthy backend still times out and records failure. Exercise the active/current path through an injected read-only PF observation seam rather than depending on the host PF configuration.
4. Add queue-level regression coverage where practical to prove the prepared-only healthy reconciliation completes and releases UpdateLock promptly, rather than holding it for the readiness timeout. Preserve the existing queue-wide lock lifetime and coalescing semantics.
5. Verify with focused daemon Gateway/reconciliation tests and CLI status/doctor tests, then run cargo fmt --all --check and cargo clippy --workspace --all-targets --all-features --locked -- -D warnings.

Task list (3/3 completed)

• Read issue, repository guidance, and relevant architecture context
• Trace gateway readiness and PF mutation locking behavior
• Assess actionability and produce an implementation plan

  ^{｜ Implement plan ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using GPT Sol ｜ 𝕏}