pv status keeps historical and superseded failures in current system health

[munezaclovis]

Summary

pv status and pv doctor treat every failed job among the last 100 records as a current whole-system failure. Plain status output omits the stored timestamps. A transient failure can therefore make PV appear broken for days—even after the underlying operation succeeds or the system is repaired.

Stale runtime observations compound the problem because a successful privileged repair such as pv ports:install does not necessarily produce a newer Gateway observation.

Verified incident behavior

The live database had 18 total jobs. Seven historical failures included:

• two expected lock conflicts during the real app update;
• multiple 60-second Gateway readiness failures caused by missing PF routing;
• one reconciliation abandoned during daemon restart;
• one transient pv.db-wal race.

After pv ports:install restored ai.test to HTTP 200 and pv update --check reported everything current:

• pv status --json still returned overall: failed;
• the old failed Gateway observation remained current in persisted runtime state;
• every historical failed job still appeared under recent_errors;
• pv doctor failed Recent jobs and recommended broad pv setup.

Current code

StatusSnapshot::read filters all failed records returned by recent_jobs() and makes any non-empty result fail aggregate health: status.rs.

The database returns the most recent 100 jobs without supersession semantics: database.rs.

Plain output prints the job identity/error but omits started_at and finished_at: status.rs.

Doctor independently fails if any recent job failed: doctor.rs.

Expected behavior

Historical failures should remain available as diagnostics, but aggregate current health should reflect unresolved/current failures rather than the mere existence of any failure in a fixed-size history window.

The exact supersession/recency policy should be made explicit before implementation.

Acceptance criteria

• Define current-versus-historical failure semantics in DESIGN.md.
• A newer successful operation or verified repaired current state can supersede an older failure for aggregate health.
• Superseded failures remain visible in job history without forcing overall: failed.
• A genuinely unresolved latest failure still produces non-zero status/doctor results.
• Plain output includes enough timestamp/age context to prevent old errors from looking current.
• A successful focused repair either refreshes affected observations or clearly marks older observations as stale/history.
• Status and doctor apply the same current-health policy.
• Add integration coverage for failure → repair/success → healthy aggregate status while preserving historical diagnostics.

Related

• pv update incurs a 10-second launchd throttle after switching app binary #300
• Full incident evidence

[pullfrog]

Assessment

This is actionable. The report matches the current implementation: status.rs and doctor.rs independently filter every failed record from Database::recent_jobs(), so retained history directly controls current health. Runtime observations are latest-only, but pv ports:install does not ask the daemon to reconcile after repairing PF, leaving an older Gateway failure current even after routing is restored.

Proposed policy

Use job scope as the supersession boundary:

• A failed terminal job is current only until a newer successful terminal job covers the same scope.
• An exact scope covers itself. A successful system job also covers older project:* and resource:* failures because system reconciliation evaluates the full desired state. Narrower jobs do not supersede a system failure.
• Running jobs do not supersede failures until they succeed.
• Superseded jobs remain unchanged and visible through pv jobs; only unresolved failures appear in the current-error summaries used by pv status and pv doctor.
• Latest runtime and Project observations remain independent current-health inputs. A successful focused repair that can invalidate one of those observations must trigger reconciliation rather than deleting or assuming health.

This avoids a schema migration, preserves the existing 100-job history, and does not let unrelated successful work hide a failure.

Plan

1. Document current-health semantics in DESIGN.md. Clarify that retained jobs are history, define scope coverage and supersession, state that only unresolved job failures affect aggregate health, and require focused integration repairs to refresh affected observed state. Verify the wording remains consistent with the latest-only observed-state contract and pv jobs history contract.

2. Add one shared state query for unresolved failures. Extend crates/state/src/database.rs with a typed query/helper derived from the existing recent jobs, preserving deterministic started_at, id ordering. Cover failure-only history, same-scope failure then success, unrelated-scope success, system-success coverage, narrower-success non-coverage, and intervening running jobs in crates/state/tests/state_foundation.rs.

3. Apply the shared policy to both diagnostics commands. Update crates/cli/src/commands/status.rs and doctor.rs to consume the shared unresolved-failure result instead of independently filtering all failures. Keep full history in pv jobs; make status JSON recent_errors represent unresolved current errors, make plain status include finished_at for each displayed failure, and make doctor’s Recent jobs result/count explicitly refer to unresolved failures. Update the nearby insta snapshots in crates/cli/tests/status.rs and doctor.rs for both unresolved-latest failure and failure followed by successful repair.

4. Refresh Gateway state after PF repair. After crates/cli/src/commands/ports.rs successfully verifies installed/active redirects, have it request and wait for a system reconciliation when the daemon is available. This produces a newer Gateway observation and a covering system-job outcome before reporting complete; if the daemon is unavailable, retain the completed PF repair but report that reconciliation could not run, while daemon health remains independently failed. Extend crates/cli/tests/ports.rs with the existing fake-daemon/socket style to verify successful repair requests reconciliation and that failed or unavailable refreshes are surfaced without rolling back valid PF files.

5. Verify the incident sequence end to end. Add integration coverage for failed system reconciliation plus failed Gateway observation, successful PF repair/reconciliation, then healthy pv status and pv doctor, while pv jobs still contains the historical failure with timestamps. Run focused state and CLI tests with cargo nextest, accept/review affected insta snapshots, then run cargo fmt --all --check and cargo clippy --workspace --all-targets --all-features --locked -- -D warnings.

Task list (4/4 completed)

• Read issue discussion, repository guidance, and relevant learnings
• Trace current status health collection, persistence, and rendering behavior
• Assess actionability and produce a verified implementation plan
• Publish the plan to issue pv status keeps historical and superseded failures in current system health #307

  ^{｜ Implement plan ➔ ｜ View workflow run ｜ via Pullfrog ｜ Using GPT Sol ｜ 𝕏}