[action] [PR:709] [module_base]: Add 'recovery' transition type to _TRANSITION_TIMEOUT_DEFAULTS#726
Merged
Conversation
…DEFAULTS
#### Description
Add a `"recovery": 600` (10 min) entry to `ModuleBase._TRANSITION_TIMEOUT_DEFAULTS`.
`set_module_state_transition()` validates the requested `transition_type` against the keys of `_TRANSITION_TIMEOUT_DEFAULTS`. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock via `set_module_state_transition(name, 'recovery')`; because `'recovery'` was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery.
#### Motivation and Context
The `'recovery'` transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in `_TRANSITION_TIMEOUT_DEFAULTS`, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829.
`recovery` has no `platform.json` override (only `startup`/`shutdown`/`reboot`/`halt_services` are overridable in `_load_transition_timeouts()`), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared in `finally`; 600 s is ample headroom.
#### How Has This Been Tested?
- Unit tests: updated the timeout-loading tests in `tests/module_base_test.py` to expect the new key. `python3 -m pytest tests/module_base_test.py` → **115 passed**.
- On-device with both master and 202605 image: with the fix, a control-plane-down DPU was power-cycled (reboot-cause "Switch rebooted DPU") and recovered to `ready_status=true`.
#### Additional Information (Optional)
Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the `'recovery'` transition this default enables.
Signed-off-by: Sonic Build Admin <sonicbld@microsoft.com>
Collaborator
Author
|
Original PR: #709 |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Collaborator
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Add a
"recovery": 600(10 min) entry toModuleBase._TRANSITION_TIMEOUT_DEFAULTS.set_module_state_transition()validates the requestedtransition_typeagainst the keys of_TRANSITION_TIMEOUT_DEFAULTS. chassisd's DPU power-cycle auto-recovery acquires the module state-transition lock viaset_module_state_transition(name, 'recovery'); because'recovery'was missing from the defaults, every recovery acquisition was rejected ("Cannot acquire state transition lock") and the power-cycle was silently skipped — blocking DPU auto-recovery.Motivation and Context
The
'recovery'transition type is used by chassisd (sonic-platform-daemons) to power-cycle an unresponsive DPU. Without a matching entry in_TRANSITION_TIMEOUT_DEFAULTS, the transition is never accepted, so a DPU that loses its control plane is never power-cycled. This unblocks the DPU auto-recovery path in sonic-net/sonic-platform-daemons#829.recoveryhas noplatform.jsonoverride (onlystartup/shutdown/reboot/halt_servicesare overridable in_load_transition_timeouts()), so the 600 s default applies uniformly. The timeout only bounds the power-cycle critical section (pre_shutdown → admin down → admin up → post_startup), which is cleared infinally; 600 s is ample headroom.How Has This Been Tested?
tests/module_base_test.pyto expect the new key.python3 -m pytest tests/module_base_test.py→ 115 passed.ready_status=true.Additional Information (Optional)
Companion to sonic-net/sonic-platform-daemons#829 (chassisd DPU auto-recovery robustness), which issues the
'recovery'transition this default enables.Signed-off-by: Sonic Build Admin sonicbld@microsoft.com