[chassisd]: Enhance DPU auto-recovery robustness and race condition handling#829
Conversation
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
…andling - Read dpu_auto_recovery from DEVICE_METADATA instead of FEATURE table - Default to disabled when dpu_auto_recovery field is missing - Direct ManualIntervention when auto-recovery disabled (skip WaitForSelfRecovery) - PCIe detach/reattach around power-cycle for clean hardware reset - Check reset_limit before issuing power-cycle to avoid wasted resets - Acquire state transition lock during power-cycle to prevent races - Re-check transition_in_progress after lock to close TOCTOU window - Abort power-cycle if lock cannot be acquired (another operation in progress) - Unify field name: use transition_in_progress (drop legacy state_transition_in_progress) - Ignore transition_type == 'recovery' so chassisd does not suppress itself - Refactor: extract _process_single_dpu_recovery to reduce update_dpu_recovery_state size - Refactor: extract _enter_booting/_enter_ready/_enter_wait_for_self_recovery helpers - Refactor: extract _should_handle_hw_offline for offline-state guard clarity - Refactor: extract _handle_boot_timeout_expired shared by Booting and PowerCycle - Add comprehensive unit tests for all new behaviors Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Hi Vasundhara When the reboot cause is unknown, the log still reports a kernel crash as the reboot cause. |
Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
@chartsai-nvidia for 4280 sku, nvidia is reporting unknown for kernel crashes and unable to distinguish. Once it is fixed, we could tighten the chassisd code. |
I understand, at this time we also report unknown reboot cause for first boot, and it would cause additional unwanted power cycle for DPU on each first boot. This will break test flows. Until there's a clarification on the reboot cause we need to fix this issue. |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Address PR sonic-net#829 review feedback: - _should_handle_hw_offline(): gate on a Ready-only allow-list instead of an exclusion list. A stale Offline snapshot seen while still AdminDown now takes the clean restart path (-> Booting) rather than power-cycling, avoiding a `module startup` race. Fail-safe for future states. - _npu_crash_on_last_boot(): treat only "kernel panic" as a crash. "unknown" is reported on first boot and must not force a power-cycle; drop the unused NPU_CRASH_REBOOT_CAUSE_KEYWORDS constant. - Update unit tests to cover the new behavior. Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
@gpunathilell I have addressed the comments in the latest changes. |
set_module_state_transition() is an atomic compare-and-acquire under a cross-process fcntl flock, and every writer of transition_in_progress goes through that same API. So the `if not lock_acquired` abort already closes the TOCTOU window against a concurrent planned shutdown/reboot: whoever grabs the lock first wins and the other's set returns False. Once recovery holds the lock, transition_type == 'recovery', so _is_planned_transition_in_progress() returns False by construction and the subsequent re-check is unreachable in real execution (only reachable by mocking). Remove the re-check block in _enter_power_cycle_or_unrecoverable() and the test_power_cycle_aborted_when_legacy_field_set_after_lock test that only exercised it via mocking. The earlier suppression gate and the lock-not-acquired abort are retained. Signed-off-by: Vasundhara Volam <vvolam@microsoft.com>
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Hi @vvolam, I think I only have two points, everything else looks good, I want to use graceful shutdown/startup use since that automatically covers the transition lock etc, so Please update that once we have fix for this command: https://github.com/sonic-net/sonic-platform-daemons/pull/829/changes#r3475789446 |
Sure, we could modify the path once we have the fix. |
|
202605 testing results are not updated |
@vaibhavhd Tests ran on both master and 202605 images on a smartswitch. Updated the PR description with the same. |
What about this? The failures likely are due to setup/state of tb, but that also means that these cases cannot be run and change is not evaluated. |
|
Are the failing cases not relevant to the change being made here? Do you want to rerun to ensure you get a green run for these cases? |
…DEFAULTS (#726) #### Description Add a `"recovery": 600` (10 min) entry to `ModuleBase._TRANSITION_TIMEOUT_DEFAULTS`. `set_module_state_transition()` validates the requested `transition_type` against the keys of `_TRANSITION_TIMEOUT_DEFAULTS`. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock via `set_module_state_transition(name, 'recovery')`; because `'recovery'` was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery. #### Motivation and Context The `'recovery'` transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in `_TRANSITION_TIMEOUT_DEFAULTS`, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829. `recovery` has no `platform.json` override (only `startup`/`shutdown`/`reboot`/`halt_services` are overridable in `_load_transition_timeouts()`), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared in `finally`; 600 s is ample headroom. #### How Has This Been Tested? - Unit tests: updated the timeout-loading tests in `tests/module_base_test.py` to expect the new key. `python3 -m pytest tests/module_base_test.py` → **115 passed**. - On-device with both master and 202605 image: with the fix, a control-plane-down DPU was power-cycled (reboot-cause "Switch rebooted DPU") and recovered to `ready_status=true`. #### Additional Information (Optional) Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the `'recovery'` transition this default enables. Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
Failure cases were because of PSU not ok on the DPUs which is fixed by [system-health]: Honor 'psu' and 'pdb' device ignore independently in hardware checker by vvolam · … and this makes all tests run clean. |
6 participants
Description
Enhance the DPU auto-recovery state machine in chassisd to fix race conditions between concurrent operations (shutdown/reboot vs power-cycle), improve correctness of the default behavior, and refactor for maintainability.
Key changes:
dpu_auto_recoveryfromDEVICE_METADATAinstead ofFEATUREtabledpu_auto_recoveryfield is missing (safer default)ManualInterventionwhen auto-recovery is disabled (skipWaitForSelfRecovery)reset_limitbefore issuing power-cycle to avoid wasted resetstransition_in_progressafter lock acquisition to close TOCTOU windowtransition_in_progress(drop legacystate_transition_in_progress)transition_type == 'recovery'so chassisd does not suppress itself_process_single_dpu_recoveryto reduceupdate_dpu_recovery_statesize_enter_booting/_enter_ready/_enter_wait_for_self_recoverytransition helpers_should_handle_hw_offlinefor offline-state guard clarity_handle_boot_timeout_expiredshared by Booting and PowerCycle statesDepends on sonic-net/sonic-platform-common PR for
get_module_state_transition/set_module_state_transition/clear_module_state_transitionplatform API.Motivation and Context
Addresses review comments from PR #815. The DPU auto-recovery feature had several race conditions:
dpu_auto_recoveryfield defaulted to enabled, which could cause unexpected power-cycles on systems without explicit configuration.How Has This Been Tested?
Unit tests: Comprehensive unit tests added in
tests/test_dpu_auto_recovery.pycovering all state transitions, race conditions, lock acquisition failures, and edge cases.On-device testing on a 202605/master image (SmartSwitch), via the sonic-mgmt
smartswitch/platform_testssuite, which exercises the DPU failure-mode / auto-recovery scenarios:ready_status=trueManualIntervention(no power-cycle)transition_type == 'recovery'exclusion)reset_limitexceeded; admin-down/up clears itFinal results (202605/master): the end-to-end auto-recovery path was confirmed — chassisd power-cycled the failed DPU and it returned to
ready_status=true. After the accompanying fixes (deviceDEVICE_METADATA.dpu_auto_recoveryYANG leaf, themodule_baserecoverytransition-timeout default, and a recovery-timeout bump),test_dpu_failure_modesfinished 5 passed / 2 failed / 1 skipped (0 errored). Both remaining failures were environmental/test-harness — adatabasedpucontainer inactive at test start, and a config-reload case that ran on pre-fix test code — not auto-recovery regressions.Verified: No regressions on remaining DPUs (remained in Ready state throughout DPU0 destructive testing)
Additional Information (Optional)