Prevent STONITH during network switch firmware upgrades

[mpeterson]

Context

On 2026-04-08 at 06:13 IDT, both Proxmox nodes (pve-01, pve-02) were power-cycled by the HA watchdog (iDRAC STONITH) after the core rack switch (USW-EnterpriseXG-24, 10.2.0.153) auto-updated firmware from 7.2.123 → 7.4.1. The switch reboot caused a brief network outage, corosync lost quorum, and the hardware watchdogs triggered power cycles on both nodes.

Root Cause

Unifi auto-update on core network infrastructure rebooted the rack switch during production hours, causing a cascading HA cluster restart.

Remediation Steps

Immediate (manual)

• Disable auto-update on core switches in Unifi Network UI (Settings → System → Auto Update)

Infrastructure (via Ansible)

• Add a dedicated corosync ring on a separate NIC/VLAN so the cluster can survive single-switch failures
• Review and potentially increase corosync token timeout for tolerance of brief network blips
• Review watchdog timeout configuration in /etc/default/pve-ha-manager
• Document maintenance procedure: put PVE nodes in maintenance mode before planned network changes

Long-term

• Consider redundant network paths (LACP/LAG across two switches) for Proxmox cluster traffic