From 91d43dff4ba5475aded6449b222fb242a2989fed Mon Sep 17 00:00:00 2001 From: Sonic Build Admin Date: Fri, 17 Jul 2026 05:10:08 +0000 Subject: [PATCH] [testbed_health_check]: Skip BMC-irrelevant health checks Summary: This PR fixes BMC runtime health-check failures caused by checks that are not applicable to BMC testbeds. On BMC topologies, the health checker was still running: - BGP session validation via bgp_facts - Interface up-port/link-state validation This caused false failures such as: - "No such container: bgp" from bgp_facts - Non-actionable interface status failures for BMC scenarios The change aligns health-check behavior with existing BMC-specific container expectations. Fixes https://github.com/sonic-net/sonic-mgmt/issues/26277 ### Type of change - [x] Bug fix - [x] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [ ] Test case improvement ### Back port request - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 - [ ] 202512 - [ ] 202605 - [x] 202608 Tracking issue/work item for backport/cherry-pick request (GitHub issue or Microsoft ADO): N/A Failure type: other ### Approach #### What is the motivation for this PR? BMC testbeds should only run relevant health checks. Running non-BMC checks produced false negatives and blocked nightly/prepare flows. #### How did you do it? Added BMC guard conditions in testbed health check logic to skip: - BGP session state check for BMC - Interface status of up ports check for BMC BMC-specific critical container checks remain enabled. #### How did you verify/test it? - Verified code path and logging behavior for BMC skip conditions. - Confirmed script diagnostics are clean after updates. - Confirmed change directly addresses observed failure mode: bgp_facts call against missing bgp container on BMC. ``` ANSIBLE_CONFIG=ansible/ansible.cfg ANSIBLE_LIBRARY=ansible/library ANSIBLE_CONNECTION_PLUGINS=ansible/plugins/connection python3 .azure-pipelines/testbed_health_check.py -t vms76-bmc-7220-1 -i ansible/str4 -o result-tmp --tbfile testbed.yaml --log-level info /opt/venv/lib/python3.12/site-packages/ansible/plugins/loader.py:1485: UserWarning: AnsibleCollectionFinder has already been configured warnings.warn('AnsibleCollectionFinder has already been configured') /var/src/sonic-mgmt-int/.azure-pipelines/testbed_health_check.py:44: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC). return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S") 2026-07-17 01:53:06,025 testbed_health_check.py#771 INFO - Checking 2026-07-17 01:53:06,025 testbed_health_check.py#168 INFO - ======================= init_hosts starts ======================= 2026-07-17 01:53:06,482 testbed_health_check.py#184 INFO - NPU hostnames: ['str4-Nokia-7220-Th6p-1-bmc'], DPU hostnames: [] [WARNING]: Found variable using reserved name: serial 2026-07-17 01:53:27,455 testbed_health_check.py#249 INFO - ======================= init_hosts ends ======================= 2026-07-17 01:53:27,455 testbed_health_check.py#284 INFO - ======================= pre_check starts ======================= 2026-07-17 01:53:30,385 testbed_health_check.py#299 INFO - {'hostname': 'str4-Nokia-7220-Th6p-1-bmc', 'reachable': True, 'failed': False, 'ping': 'pong', 'invocation': {'module_args': {'data': 'pong'}}, 'ansible_facts': {'discovered_interpreter_python': '/usr/bin/python3.13'}, 'warnings': ['Platform linux on host str4-Nokia-7220-Th6p-1-bmc is using the discovered Python interpreter at /usr/bin/python3.13, but future installation of another Python interpreter could change the meaning of that path. See https://docs.ansible.com/ansible-core/2.18/reference_appendices/interpreter_discovery.html for more information.'], '_ansible_no_log': False, 'changed': False} 2026-07-17 01:53:33,180 testbed_health_check.py#402 INFO - ======================= pre_check ends ======================= 2026-07-17 01:53:33,180 testbed_health_check.py#699 INFO - ======================= check_critical_containers_running starts ======================= 2026-07-17 01:53:33,180 testbed_health_check.py#708 INFO - ----------------------- check_critical_containers_running on [str4-Nokia-7220-Th6p-1-bmc] ----------------------- 2026-07-17 01:53:34,657 testbed_health_check.py#746 INFO - ======================= check_critical_containers_running ends ======================= 2026-07-17 01:53:34,657 testbed_health_check.py#481 INFO - ======================= skip check_bgp_session_state for bmc ======================= 2026-07-17 01:53:34,657 testbed_health_check.py#576 INFO - =================== skip check_interface_status_of_up_ports for bmc =================== 2026-07-17 01:53:34,657 testbed_health_check.py#420 INFO - Check finished. Testbed is healthy. ``` #### Any platform specific information? Yes. This change is specific to BMC topologies and does not alter non-BMC behavior. #### Supported testbed topology if it's a new test case? N/A (no new test case added). ### Documentation No documentation update required (behavioral fix in existing health-check flow). Signed-off-by: Sonic Build Admin --- .azure-pipelines/testbed_health_check.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.azure-pipelines/testbed_health_check.py b/.azure-pipelines/testbed_health_check.py index 5eeab49c9..e4b5857d2 100644 --- a/.azure-pipelines/testbed_health_check.py +++ b/.azure-pipelines/testbed_health_check.py @@ -443,6 +443,10 @@ def check_bgp_session_state(self, state="established"): logger.info("======================= skip check_bgp_session_state for snappi =======================") return + if self.is_bmc_testbed: + logger.info("======================= skip check_bgp_session_state for bmc =======================") + return + def find_unexpected_bgp_neighbors(neigh_bgp_facts, expected_state, unexpected_neighbors): for k, v in list(neigh_bgp_facts['bgp_neighbors'].items()): if v['state'] != expected_state: @@ -534,6 +538,10 @@ def check_interface_status_of_up_ports(self): logger.info("=================== skip check_interface_status_of_up_ports for snappi ===================") return + if self.is_bmc_testbed: + logger.info("=================== skip check_interface_status_of_up_ports for bmc ===================") + return + failed = False interface_facts_on_hosts = {}