Issue Description
The Azure.sonic-sairedis PR CI is intermittently failing in VS/DVS tests. This was observed while validating sonic-net/sonic-sairedis PR #1962 (Cast workaround for N/A counter values from 202412). The PR has reportedly hit this CI failure pattern multiple times.
The failure does not appear related to the PR code path. The common pattern is that the DVS container or services fail to become ready, then later tests cascade with Redis socket and Docker container-not-found errors.
Related PR:
Example Azure build/log:
Failure summary
Pipeline:
- Azure.sonic-sairedis
- Build:
1153496 / 20260701.3
- Stage/job: VS tests (
TestAsan / vstest / Run vs tests log shows the same DVS failure signature; the linked Test / vstest job was still in progress when triaged)
Primary setup failure:
AssertionError: Operation timed out after 60 seconds with result {
'arp_update': 'STOPPED', 'bgpd': 'STOPPED', 'buffermgrd': 'STOPPED',
'coppmgrd': 'STOPPED', 'fdbsyncd': 'STOPPED', 'fpmsyncd': 'STOPPED',
'intfmgrd': 'STOPPED', 'orchagent': 'STOPPED', 'portsyncd': 'STOPPED',
'redis-server': 'RUNNING', 'rsyslogd': 'RUNNING', 'start.sh': 'EXITED',
'syncd': 'EXITED', 'zebra': 'STOPPED', ...
}
dvslib/dvs_common.py:60: AssertionError
Follow-on cleanup/setup errors:
RuntimeError: Unable to connect to redis (unix-socket) - No such file or directory(1): Cannot assign requested address
DASH tests later cascade with Docker container disappearance:
docker.errors.NotFound: 404 Client Error for http+docker://localhost/v1.54/containers/<id>/exec:
Not Found ("No such container: <id>")
Example affected tests from the log:
ERROR test_acl_inner_src_mac_rewrite.py::TestInnerSrcMacRewriteAclTable::test_InnerSrcMacRewriteAclTableCreationDeletion
ERROR test_acl_inner_src_mac_rewrite.py::TestInnerSrcMacRewriteAclTable::test_InnerSrcMacRewriteAclRuleCreationDeletion
ERROR test_acl_inner_src_mac_rewrite.py::TestInnerSrcMacRewriteAclTable::test_InnerSrcMacRewriteAclRuleUpdate
ERROR dash/test_dash_acl.py::TestAcl::test_multiple_tags_and_prefixes
ERROR dash/test_dash_meter.py::TestDashMeter::test_v4_meter
ERROR dash/test_dash_vnet.py::TestDash::test_vnet
One DASH run summary in the same log ended with:
12 passed, 1 skipped, 10 warnings, 26 errors in 572.45s
Triage notes
- Classification: flaky / infrastructure-level VS DVS setup failure.
- The first failure is DVS service readiness timeout: most services remain
STOPPED, while redis-server is RUNNING and start.sh / syncd are EXITED.
- The later
docker.errors.NotFound and Redis socket errors look like cascading cleanup/setup failures after the DVS container becomes unusable or is removed.
- This is blocking PR validation in sonic-sairedis but the failing framework/tests are from sonic-swss VS/DVS tests.
Proposed fix direction
- Investigate why the DVS container reaches
start.sh: EXITED / syncd: EXITED with most services stopped during VS test setup.
- Improve DVS setup failure diagnostics: capture container logs and service startup logs before teardown removes the container.
- Avoid cascading Docker NotFound failures masking the original DVS startup failure; teardown should tolerate an already-removed container and preserve the root-cause log.
- If this is a known transient startup race, add retry/health-check hardening around DVS startup rather than allowing unrelated test modules to fail.
Issue Description
The Azure.sonic-sairedis PR CI is intermittently failing in VS/DVS tests. This was observed while validating sonic-net/sonic-sairedis PR #1962 (
Cast workaround for N/A counter values from 202412). The PR has reportedly hit this CI failure pattern multiple times.The failure does not appear related to the PR code path. The common pattern is that the DVS container or services fail to become ready, then later tests cascade with Redis socket and Docker container-not-found errors.
Related PR:
Example Azure build/log:
Failure summary
Pipeline:
1153496/20260701.3TestAsan / vstest / Run vs testslog shows the same DVS failure signature; the linkedTest / vstestjob was still in progress when triaged)Primary setup failure:
Follow-on cleanup/setup errors:
DASH tests later cascade with Docker container disappearance:
Example affected tests from the log:
One DASH run summary in the same log ended with:
Triage notes
STOPPED, whileredis-serverisRUNNINGandstart.sh/syncdareEXITED.docker.errors.NotFoundand Redis socket errors look like cascading cleanup/setup failures after the DVS container becomes unusable or is removed.Proposed fix direction
start.sh: EXITED/syncd: EXITEDwith most services stopped during VS test setup.