tests: harden DVS startup failure handling#4729
Open
StormLiangMS wants to merge 2 commits into
Open
Conversation
Increase the default DVS service readiness wait to reduce CI flakes under load, keep it configurable via DVS_SERVICE_READY_TIMEOUT, and avoid retaining a destroyed DVS object after failed recreation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Storm Liang <stormliang@microsoft.com>
Collaborator
|
/azp run |
This was referenced Jul 2, 2026
Closed
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR hardens the VS/DVS pytest harness in sonic-swss to reduce flaky CI failures during DVS startup by extending the service-readiness wait and making DVS lifecycle handling more resilient after startup failures.
Changes:
- Increased default DVS service readiness timeout from 60s to 180s, with an environment-variable override (
DVS_SERVICE_READY_TIMEOUT). - Cleared the session-scoped
dvsreference after destroy so failed recreations don’t leave a stale destroyed object. - Skipped session teardown when no
dvsobject exists to avoid cascading cleanup errors after startup failures.
| FABRIC_NUM_PORTS = 16 | ||
|
|
||
| SINGLE_ASIC_VOQ_FS = "single_asic_voq_fs" | ||
| DEFAULT_DVS_SERVICE_READY_TIMEOUT = int(os.environ.get("DVS_SERVICE_READY_TIMEOUT", "180")) |
Fall back to the default DVS service readiness timeout when DVS_SERVICE_READY_TIMEOUT is set to an invalid or empty value, so pytest collection is not broken by malformed environment input. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Storm Liang <stormliang@microsoft.com>
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Contributor
Author
|
Addressed Copilot review feedback in ca03698: DVS_SERVICE_READY_TIMEOUT parsing is now guarded and falls back to 180 when the environment variable is empty or non-numeric, so pytest collection will not break on malformed input. |
Contributor
|
@StormLiangMS there were some DVS image changes were previously causing VS test issues, these should be fixed by this buildimage PR: sonic-net/sonic-buildimage#28108. Please check if you are still seeing issues. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Azure.sonic-sairedis PR CI intermittently fails in VS/DVS setup. The failure pattern tracked in #4728 shows DVS service readiness timing out while most services remain STOPPED and
start.sh/syncdexit. After a failed recreate, follow-on tests can cascade with Redis socket and DockerNo such containererrors.What changed
DVS_SERVICE_READY_TIMEOUT.NoneType/destroyed-container teardown errors after startup failure.Validation
python -m py_compile tests/conftest.py tests/dvslib/dvs_common.pygit diff --checkRelated