Global reconciliation lock is reported as “PV update is already in progress”

[munezaclovis]

Summary

PV's UpdateLock is also the daemon's global lock for queued mutating reconciliation work. When reconciliation owns it, blocked commands still report:

PV update is already in progress; update lock is held at ~/.pv/run/update.lock

This makes a long reconciliation or readiness probe look like a stuck app update.

Verified incident behavior

A Project reconciliation held the lock from July 28 21:10:56 to 21:11:57 while Gateway readiness waited on unreachable public ports. During that minute, pv update --check would accurately detect a live kernel lock but misidentify its owner as an app update.

The lock was not held for a day. Once reconciliation ended:

• the empty lock file remained on disk, as designed;
• no process had it open;
• pv update --check succeeded immediately.

Historical lock errors also continued appearing in pv status, making the wording more misleading.

Current code

The error is permanently update-specific: StateError::UpdateInProgress.

The daemon reconciliation queue acquires the same UpdateLock for mutating work: reconciliation.rs.

The lock implementation correctly uses an OS flock; the file's continued existence is not itself treated as ownership: update_lock.rs.

Expected behavior

The error should describe the actual concurrency condition without falsely claiming an app update owns the lock.

A minimal compatible fix may use neutral wording such as “another PV mutating operation is in progress.” Rich owner/operation metadata can be considered separately and is not required to fix the false statement.

Acceptance criteria

• A reconciliation-owned lock is not reported as an app update.
• App-update flows may retain update-specific wording when ownership is actually known.
• The message clearly distinguishes a live OS lock from a stale on-disk lock file.
• Existing mutual exclusion and lock lifetime behavior remain unchanged.
• CLI, daemon job errors, snapshots, and JSON-visible errors use consistent terminology.
• Add coverage for both app-update-owned and reconciliation-owned contention.
• Keep operation-owner metadata out of scope unless required by the chosen design.

Related

• pv update incurs a 10-second launchd throttle after switching app binary #300
• Full incident evidence

[pullfrog]

Assessment

This issue is actionable. UpdateLock is a live, advisory OS lock rather than a lock-file existence check, but both the foreground app-update phase and the daemon reconciliation queue acquire it. Because the lock contains no owner metadata, a contender cannot truthfully identify the holder as an app update. The minimal fix is neutral wording for all contention; adding owner metadata is unnecessary and remains out of scope.

Plan

1. Update the lock contract in DESIGN.md to document that ~/.pv/run/update.lock is the compatibility path for PV’s global mutating-operation lock, that foreground app updates and queued daemon mutations share it, and that an unlocked file may remain on disk. Preserve the current acquisition and lifetime rules.
2. Replace the update-specific contention errors in crates/state/src/error.rs and crates/cli/src/error.rs with neutral mutating-operation terminology, and update the mappings in crates/cli/src/commands/mod.rs, crates/cli/src/commands/update.rs, and crates/daemon/src/jobs.rs. Use wording such as another PV mutating operation is in progress; the live OS lock is held at {path} so the message does not infer an owner and does not imply that file existence means contention.
3. Keep UpdateLock, the update.lock path, OS flock implementation, queue reservation, and release behavior unchanged. Rename only error/helper identifiers where needed to prevent update-specific semantics from leaking into future call sites; do not add operation-owner metadata.
4. Extend the existing state and daemon tests to cover both directions: an app-update-held lock rejecting daemon mutation, and a reconciliation-reserved lock rejecting another lock probe. Assert the neutral typed error and its rendered text while retaining the stale-file/reacquisition coverage in crates/state/tests/state_foundation.rs and the queue lifetime coverage in crates/daemon/src/jobs.rs.
5. Refresh the affected CLI and daemon insta snapshots, including blocked pv update, pv update --check, blocked non-update commands, daemon response JSON, and persisted/forwarded daemon error text, so every human-readable and JSON-visible surface uses the same terminology.
6. Verify with cargo nextest run -p state -p daemon -p cli --locked, cargo fmt --all --check, and cargo clippy --workspace --all-targets --all-features --locked -- -D warnings.

Task list (4/4 completed)

• Read issue, repository guidance, and relevant durable learnings
• Trace reconciliation locking and user-facing update messaging
• Determine actionability and produce an implementation plan
• Publish the plan to issue Global reconciliation lock is reported as “PV update is already in progress” #308

  ^{｜ Implement plan ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using GPT Sol ｜ 𝕏}