Summary
Since #25062 enabled pc/test_po_cleanup.py on sonic-vpp, test_po_cleanup_after_reload intermittently ERRORs at teardown on the t1-lag-vpp KVM testbed. Both test bodies pass — the failure is the global memory_utilization plugin failing the teardown because monit reports a transient memory increase during the test's CPU stress + config_reload.
These tests were enabled on sonic-vpp by #25062 ("chore: enable pc VPP tests p1", commit cbaaa8bdd, 2026-06-24, @cyw233), which removed the sonic-vpp skip markers for pc/test_po_cleanup.py::test_po_cleanup and ::test_po_cleanup_after_reload.
Failure signature
Build 1152688, DUT vlab-vpp-01:
============= 2 passed, 162 warnings, 1 error in 643.23s (0:10:43) =============
ERROR at teardown of test_po_cleanup_after_reload[vlab-vpp-01]
E Failed: [ALARM]: monit:memory_usage memory usage increased by 20.9%,
exceeds increase threshold 10% (previous: 19.9%, current: 40.8%)
common/plugins/memory_utilization/__init__.py:206: Failed
Elastictest test plan: https://elastictest.org/scheduler/publictestplan/6a439f9cf9d3963c406e2b07?testcase=pc%2Ftest_po_cleanup.py&type=console
DUT facts (from the run's conditional-skip dump): asic_type=vpp, asic_subtype=vpp, platform=x86_64-kvm_x86_64-r0, topo_name=t1-lag, is_multi_asic=false, pytest 9.1.1.
Root cause
test_po_cleanup_after_reload deliberately drives CPU to 100% (yes processes) and runs config_reload, which transiently spikes memory usage. On the low-RAM sonic-vpp KVM VM this briefly exceeds the memory_utilization plugin's 10% increase threshold, so the autouse fixture calls pytest.fail() at teardown (common/plugins/memory_utilization/__init__.py:206). The po_cleanup logic itself passes — the failure is the memory-delta check reacting to the reload/stress transient, not a po_cleanup functional issue.
Observed scope
test_po_cleanup → PASSES
test_po_cleanup_after_reload → body passes, ERROR at teardown (flaky — only when the transient spike crosses the threshold)
- Elastictest aggregate:
P1 / S0 / F1
Proposed fix
Add @pytest.mark.disable_memory_utilization to test_po_cleanup_after_reload — the established pattern for reboot/reload/stress tests (e.g. platform_tests/test_reboot.py, golden_config_infra/test_config_reload_with_rendered_golden_config.py, upgrade_path/test_warmboot_data_consistency.py). The test still runs and validates real po_cleanup behavior; only the memory-delta teardown check is skipped for this reload/stress test.
Fix PR: #25820
Impacted PRs (t1-lag-vpp CI red on this)
cc @cyw233
Summary
Since #25062 enabled
pc/test_po_cleanup.pyon sonic-vpp,test_po_cleanup_after_reloadintermittently ERRORs at teardown on thet1-lag-vppKVM testbed. Both test bodies pass — the failure is the globalmemory_utilizationplugin failing the teardown becausemonitreports a transient memory increase during the test's CPU stress +config_reload.These tests were enabled on sonic-vpp by #25062 ("chore: enable pc VPP tests p1", commit
cbaaa8bdd, 2026-06-24, @cyw233), which removed the sonic-vpp skip markers forpc/test_po_cleanup.py::test_po_cleanupand::test_po_cleanup_after_reload.Failure signature
Build
1152688, DUTvlab-vpp-01:Elastictest test plan: https://elastictest.org/scheduler/publictestplan/6a439f9cf9d3963c406e2b07?testcase=pc%2Ftest_po_cleanup.py&type=console
DUT facts (from the run's conditional-skip dump):
asic_type=vpp,asic_subtype=vpp,platform=x86_64-kvm_x86_64-r0,topo_name=t1-lag,is_multi_asic=false, pytest9.1.1.Root cause
test_po_cleanup_after_reloaddeliberately drives CPU to 100% (yesprocesses) and runsconfig_reload, which transiently spikes memory usage. On the low-RAM sonic-vpp KVM VM this briefly exceeds thememory_utilizationplugin's 10% increase threshold, so the autouse fixture callspytest.fail()at teardown (common/plugins/memory_utilization/__init__.py:206). The po_cleanup logic itself passes — the failure is the memory-delta check reacting to the reload/stress transient, not a po_cleanup functional issue.Observed scope
test_po_cleanup→ PASSEStest_po_cleanup_after_reload→ body passes, ERROR at teardown (flaky — only when the transient spike crosses the threshold)P1 / S0 / F1Proposed fix
Add
@pytest.mark.disable_memory_utilizationtotest_po_cleanup_after_reload— the established pattern for reboot/reload/stress tests (e.g.platform_tests/test_reboot.py,golden_config_infra/test_config_reload_with_rendered_golden_config.py,upgrade_path/test_warmboot_data_consistency.py). The test still runs and validates real po_cleanup behavior; only the memory-delta teardown check is skipped for this reload/stress test.Fix PR: #25820
Impacted PRs (t1-lag-vpp CI red on this)
impacted-area-kvmtest-t1-lag-vppselectspc/test_po_cleanup.py)cc @cyw233