Harden the external reliability controller

[JROChub]

What changed

• moved the production soak controller to a dedicated hardened system service
• parallelized public probes, validator audits, hourly bursts, and status publication
• added bounded retries with raw attempt evidence and confirmed-error accounting
• added a strict ten-sample preflight that uses ephemeral state
• expanded regression coverage for transient and exhausted RPC/SSH failures
• documented the dedicated external-controller topology and operating procedure

Root cause

The previous controller ran on an operator Wi-Fi workstation. A local route and DNS interruption made every external probe fail simultaneously. Sequential SSH audits and status publication then extended that controller-side interruption across multiple sample intervals. Single transport timeouts were also recorded immediately as permanent RPC failures.

Validation

• python3 scripts/test_reliability_campaign.py
• python3 scripts/test_status_registry.py
• python3 scripts/test_observer_registry.py
• python3 scripts/test_observer_intake.py
• shellcheck scripts/install_reliability_campaign.sh
• systemd-analyze verify infra/systemd/powerhouse-reliability-controller.service
• live ten-sample cross-region preflight: 10/10 clean, zero retries
• live campaign browser render: ON TRACK, 100% uptime, 3/3 validators, 1/1 observer
• all three published campaign replicas have identical SHA-256 state

Deployment

The dedicated Toronto controller is live. The failed workstation campaign was disabled and archived without deleting its hash-chained evidence. The official replacement campaign began at 2026-07-10T19:17:45Z.