[pc/test_po_cleanup] Skip memory_utilization for test_po_cleanup_after_reload by AnantKishorSharma · Pull Request #25820 · sonic-net/sonic-mgmt

AnantKishorSharma · 2026-07-01T16:19:33Z

Why I did it

pc/test_po_cleanup.py::test_po_cleanup_after_reload intermittently ERRORs at teardown on the t1-lag-vpp KVM testbed. The test body passes; the failure is the global memory_utilization plugin failing the teardown because monit reports a transient memory increase during the test's CPU stress + config_reload:

ERROR at teardown of test_po_cleanup_after_reload[vlab-vpp-01]
E   Failed: [ALARM]: monit:memory_usage memory usage increased by 20.9%,
    exceeds increase threshold 10% (previous: 19.9%, current: 40.8%)
common/plugins/memory_utilization/__init__.py:206: Failed

These tests were enabled on sonic-vpp by #25062. On the low-RAM vpp KVM VM the reload/stress transient crosses the plugin's 10% memory-increase threshold, even though the po_cleanup behavior is correct.

Fixes #25819

How I did it

Add vpp-scoped conditional-mark YAML entry. The test still runs and validates real po_cleanup behavior; only the memory-delta teardown check is skipped for it. test_po_cleanup is unchanged.

How to verify it

Re-run pc/test_po_cleanup.py on t1-lag-vpp; test_po_cleanup_after_reload no longer ERRORs at teardown of the memory_utilization fixture (common/plugins/memory_utilization/__init__.py:206).

Unblocks: sonic-net/sonic-buildimage#28164, sonic-net/sonic-buildimage#28042

…r_reload test_po_cleanup_after_reload drives CPU to 100% and runs config_reload, which transiently spikes memory usage. On the low-memory sonic-vpp KVM VM this briefly trips the monit memory_usage alarm and fails the global memory_utilization teardown check, even though the test body itself passes (2 passed, 1 error at teardown). Add @pytest.mark.disable_memory_utilization to this reload/stress test, matching other reboot/reload tests. test_po_cleanup is unaffected. Signed-off-by: Anant Kishor Sharma <anant.kishor-sharma@hpe.com>

mssonicbld · 2026-07-01T16:19:41Z

/azp run

azure-pipelines · 2026-07-01T16:19:57Z

Azure Pipelines successfully started running 1 pipeline(s).

saiarcot895 · 2026-07-01T16:25:48Z

+# On the low-memory sonic-vpp KVM VM that briefly trips the monit memory_usage alarm and fails
+# the global memory_utilization teardown check, even though the test body itself passes. Skip
+# that check for this reload/stress test (consistent with other reboot/reload/stress tests).
+@pytest.mark.disable_memory_utilization


Is it possible to disable this only for VPP?

Alternatively, if it is a transient spike, would it make sense to sleep for a few seconds in this test before exiting out, so that the memory usage check can still be preserved?

Thanks! Scoped to VPP — dropped the test marker and applied disable_memory_utilization via tests_mark_conditions_sonic_vpp.yaml (asic_type in ['vpp']), same mechanism as skip_traffic_test, so the check stays active on all other platforms.

On the sleep: a short sleep likely won't reliably clear the 10% threshold on the low-RAM KVM VM — but happy to try that if you prefer.

…up_after_reload Signed-off-by: Anant Kishor Sharma <anant.kishor-sharma@hpe.com>

mssonicbld · 2026-07-01T16:48:00Z

/azp run

github-actions · 2026-07-01T16:48:14Z

@StormLiangMS @wangxin @yxieca A user wants to merge changes to the conditional mark files into master. Please review.

azure-pipelines · 2026-07-01T16:48:15Z

Azure Pipelines successfully started running 1 pipeline(s).

yxieca

Approving: well-scoped, correctly-justified disable of an infra teardown guard.

The memory_utilization teardown check is a relative (>10%) memory-increase guard; on the low-RAM t1-lag-vpp KVM it trips transiently during config_reload + CPU stress (19.9% -> 40.8%) even though the test_po_cleanup_after_reload body passes. Disabling only that teardown check for this one test, scoped to asic_type in ['vpp'] via the conditional-mark YAML, is the right, minimal fix: non-vpp platforms (and all other tests on vpp) still enforce the memory-delta check, and po_cleanup behavior is still validated. This is consistent with the established disable_memory_utilization pattern (test_reboot, golden_config, warmboot, and others), and is actually tighter than that prior art since it is platform-scoped rather than a global source marker.

One minor follow-up (non-blocking): the PR description says a @pytest.mark.disable_memory_utilization was added to the test, but the change is actually the vpp-scoped conditional-mark YAML entry. Updating the description to match would help future readers/git blame.

Reviewed by AI agent on behalf of Ying.

…r_reload (sonic-net#25820) What: Adds a disable_memory_utilization conditional-mark entry for pc/test_po_cleanup.py::test_po_cleanup_after_reload (asic_type in ['vpp']) in tests/common/plugins/conditional_mark/tests_mark_conditions_sonic_vpp.yaml. Why: Fixes sonic-net#25819 — on the low-RAM sonic-vpp KVM VM, config_reload with CPU stress transiently pushes monit memory_usage past the plugin's 10% increase threshold, ERRORing at teardown even though the test body passes. How: Applies the established @disable_memory_utilization pattern (as used by reboot/reload/stress tests) scoped to vpp only; the test still runs and validates po_cleanup behavior, only the memory-delta teardown check is skipped. test_po_cleanup is unchanged. Testing: All Azure KVM jobs (t0/t1-lag/t1-lag-vpp/t2/dpu/dualtor/multi-asic/2vlans/sonic), CodeQL, Semgrep, DCO, EasyCLA passing. Approved by saiarcot895. Signed-off-by: Anant Kishor Sharma <anant.kishor-sharma@hpe.com>

…r_reload (sonic-net#25820) What: Adds a disable_memory_utilization conditional-mark entry for pc/test_po_cleanup.py::test_po_cleanup_after_reload (asic_type in ['vpp']) in tests/common/plugins/conditional_mark/tests_mark_conditions_sonic_vpp.yaml. Why: Fixes sonic-net#25819 — on the low-RAM sonic-vpp KVM VM, config_reload with CPU stress transiently pushes monit memory_usage past the plugin's 10% increase threshold, ERRORing at teardown even though the test body passes. How: Applies the established @disable_memory_utilization pattern (as used by reboot/reload/stress tests) scoped to vpp only; the test still runs and validates po_cleanup behavior, only the memory-delta teardown check is skipped. test_po_cleanup is unchanged. Testing: All Azure KVM jobs (t0/t1-lag/t1-lag-vpp/t2/dpu/dualtor/multi-asic/2vlans/sonic), CodeQL, Semgrep, DCO, EasyCLA passing. Approved by saiarcot895. Signed-off-by: Anant Kishor Sharma <anant.kishor-sharma@hpe.com> Signed-off-by: selldinesh <dinesh.sellappan@keysight.com>

…larms (#25911)  ### Description of PR  Summary: The autouse `memory_utilization` fixture fails a test in teardown when a monitored process's memory grows past a threshold. On the **vpp** KVM baseline this fires spuriously and shows up as a teardown `ERROR` even though the test body passed. FRR starts near-empty on that baseline, so any test that brings BGP up/down (for example `ssh/test_ssh_stress`, which runs `config bgp startup/shutdown all`) grows `bgpd`/`zebra` far past the hardware-calibrated thresholds: ``` ERROR at teardown of test_ssh_stress[vlab-vpp-01] [ALARM]: frr_bgp:used memory usage increased by 250.0% ... (previous: 28.0 MB, current: 98.0 MB) [ALARM]: top:zebra memory usage increased by 195.9 MB ... (previous: 47.8 MB, current: 243.7 MB) ``` The plugin already downgrades these alarms to warnings on the **vs** virtual platform, and #23359 relaxed the `free` check on **vpp** for the same stated reason (*"vpp dataplane runs in host"*). This change completes that carve-out: `vpp` is a virtual/software platform whose process/FRR memory is not hardware-representative, so treat it like `vs` -- log memory alarms as warnings for visibility, but do not fail the test. Real hardware is unchanged (memory-leak gating is preserved where it is meaningful) and `vs` is unchanged. This also removes the need for per-test `disable_memory_utilization` skips on vpp (e.g. #25820). Fixes: stops the `memory_utilization` fixture from failing vpp tests at teardown on benign, non-hardware-representative memory growth, by treating vpp like vs (log memory alarms as warnings instead of failing).  ### Type of change  - [x] Bug fix - [x] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [ ] Test case improvement ### Back port request  - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 - [ ] 202512 - [ ] 202605 Tracking issue/work item for backport/cherry-pick request: Failure type: day-one issue ### Approach #### What is the motivation for this PR? The memory-leak check is valuable on real hardware, but on the virtual `vpp` baseline it produces false failures: memory footprints there are not hardware-representative (the dataplane runs in the host and FRR starts near-empty), so hardware-calibrated thresholds are tripped by benign, expected growth. Those false teardown errors show up as flaky nightly/PR failures across any vpp test that exercises BGP. #### How did you do it? At the single alarm choke point (`_handle_memory_threshold_exceeded`), extend the existing virtual-platform handling so `vpp` is treated the same as `vs`: log the alarm as a warning (for visibility) instead of storing an error that fails the test in the fixture teardown. This is one condition change (`asic_type in ("vs", "vpp")`) plus an explanatory comment; no thresholds and no other platform behavior change. Because the pre-commit `flake8` hook lints the whole touched file, it also surfaced 4 pre-existing `E231` false positives on f-string format specs (`f"{val:.1f}"`, a flake8-6.0.0/py3.12 quirk). These are suppressed with `# noqa: E231`, matching the suppression style already used in this file (e.g. `# noqa: E501`). No behavior change. #### How did you verify/test it? Elastictest, `ssh/test_ssh_stress.py` (which brings BGP up/down and trips the alarm) repeated 20x on the vpp KVM baseline (`vms-kvm-vpp-t1-lag`, t1-lag-vpp): - **Before (old code -- alarm fails the test):** https://elastictest.org/scheduler/testplan/6a47f6fda577b9de602ff366 -- 4/20 runs error at teardown, e.g. `[ALARM]: frr_bgp:used memory usage increased by 250.0% ... (28 -> 98 MB)`, where the `memory_utilization` fixture calls `pytest.fail` (the test body itself passes). - **After (this PR -- alarm is a warning):** https://elastictest.org/scheduler/testplan/6a4b43f5d38c80f6ad94be68 -- 20/20 pass; the same `frr_bgp`/`zebra` alarm is logged as a warning and no longer errors the test at teardown. Also: - `flake8 --max-line-length=120` clean (matches CI); `python -m py_compile` clean. #### Any platform specific information? Only the `vpp` asic path changes (its memory alarms become warnings). `vs` was already handled this way; physical-hardware platforms are unchanged and continue to fail on memory-threshold violations. #### Supported testbed topology if it's a new test case? N/A (framework plugin change, not a new test case). ### Documentation  No documentation changes. Signed-off-by: Augustine Lee <augustinelee@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…r_reload (sonic-net#25820) What: Adds a disable_memory_utilization conditional-mark entry for pc/test_po_cleanup.py::test_po_cleanup_after_reload (asic_type in ['vpp']) in tests/common/plugins/conditional_mark/tests_mark_conditions_sonic_vpp.yaml. Why: Fixes sonic-net#25819 — on the low-RAM sonic-vpp KVM VM, config_reload with CPU stress transiently pushes monit memory_usage past the plugin's 10% increase threshold, ERRORing at teardown even though the test body passes. How: Applies the established @disable_memory_utilization pattern (as used by reboot/reload/stress tests) scoped to vpp only; the test still runs and validates po_cleanup behavior, only the memory-delta teardown check is skipped. test_po_cleanup is unchanged. Testing: All Azure KVM jobs (t0/t1-lag/t1-lag-vpp/t2/dpu/dualtor/multi-asic/2vlans/sonic), CodeQL, Semgrep, DCO, EasyCLA passing. Approved by saiarcot895. Signed-off-by: Anant Kishor Sharma <anant.kishor-sharma@hpe.com> Signed-off-by: ssithaia-ebay <ssithaian@ebay.com>

…larms (sonic-net#25911)  ### Description of PR  Summary: The autouse `memory_utilization` fixture fails a test in teardown when a monitored process's memory grows past a threshold. On the **vpp** KVM baseline this fires spuriously and shows up as a teardown `ERROR` even though the test body passed. FRR starts near-empty on that baseline, so any test that brings BGP up/down (for example `ssh/test_ssh_stress`, which runs `config bgp startup/shutdown all`) grows `bgpd`/`zebra` far past the hardware-calibrated thresholds: ``` ERROR at teardown of test_ssh_stress[vlab-vpp-01] [ALARM]: frr_bgp:used memory usage increased by 250.0% ... (previous: 28.0 MB, current: 98.0 MB) [ALARM]: top:zebra memory usage increased by 195.9 MB ... (previous: 47.8 MB, current: 243.7 MB) ``` The plugin already downgrades these alarms to warnings on the **vs** virtual platform, and sonic-net#23359 relaxed the `free` check on **vpp** for the same stated reason (*"vpp dataplane runs in host"*). This change completes that carve-out: `vpp` is a virtual/software platform whose process/FRR memory is not hardware-representative, so treat it like `vs` -- log memory alarms as warnings for visibility, but do not fail the test. Real hardware is unchanged (memory-leak gating is preserved where it is meaningful) and `vs` is unchanged. This also removes the need for per-test `disable_memory_utilization` skips on vpp (e.g. sonic-net#25820). Fixes: stops the `memory_utilization` fixture from failing vpp tests at teardown on benign, non-hardware-representative memory growth, by treating vpp like vs (log memory alarms as warnings instead of failing).  ### Type of change  - [x] Bug fix - [x] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [ ] Test case improvement ### Back port request  - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 - [ ] 202512 - [ ] 202605 Tracking issue/work item for backport/cherry-pick request: Failure type: day-one issue ### Approach #### What is the motivation for this PR? The memory-leak check is valuable on real hardware, but on the virtual `vpp` baseline it produces false failures: memory footprints there are not hardware-representative (the dataplane runs in the host and FRR starts near-empty), so hardware-calibrated thresholds are tripped by benign, expected growth. Those false teardown errors show up as flaky nightly/PR failures across any vpp test that exercises BGP. #### How did you do it? At the single alarm choke point (`_handle_memory_threshold_exceeded`), extend the existing virtual-platform handling so `vpp` is treated the same as `vs`: log the alarm as a warning (for visibility) instead of storing an error that fails the test in the fixture teardown. This is one condition change (`asic_type in ("vs", "vpp")`) plus an explanatory comment; no thresholds and no other platform behavior change. Because the pre-commit `flake8` hook lints the whole touched file, it also surfaced 4 pre-existing `E231` false positives on f-string format specs (`f"{val:.1f}"`, a flake8-6.0.0/py3.12 quirk). These are suppressed with `# noqa: E231`, matching the suppression style already used in this file (e.g. `# noqa: E501`). No behavior change. #### How did you verify/test it? Elastictest, `ssh/test_ssh_stress.py` (which brings BGP up/down and trips the alarm) repeated 20x on the vpp KVM baseline (`vms-kvm-vpp-t1-lag`, t1-lag-vpp): - **Before (old code -- alarm fails the test):** https://elastictest.org/scheduler/testplan/6a47f6fda577b9de602ff366 -- 4/20 runs error at teardown, e.g. `[ALARM]: frr_bgp:used memory usage increased by 250.0% ... (28 -> 98 MB)`, where the `memory_utilization` fixture calls `pytest.fail` (the test body itself passes). - **After (this PR -- alarm is a warning):** https://elastictest.org/scheduler/testplan/6a4b43f5d38c80f6ad94be68 -- 20/20 pass; the same `frr_bgp`/`zebra` alarm is logged as a warning and no longer errors the test at teardown. Also: - `flake8 --max-line-length=120` clean (matches CI); `python -m py_compile` clean. #### Any platform specific information? Only the `vpp` asic path changes (its memory alarms become warnings). `vs` was already handled this way; physical-hardware platforms are unchanged and continue to fail on memory-threshold violations. #### Supported testbed topology if it's a new test case? N/A (framework plugin change, not a new test case). ### Documentation  No documentation changes. Signed-off-by: Augustine Lee <augustinelee@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: ssithaia-ebay <ssithaian@ebay.com>

…'t fail) on memory alarms (#26510)  ### Description of PR  Summary: The autouse `memory_utilization` fixture fails a test in teardown when a monitored process's memory grows past a threshold. On the **vpp** KVM baseline this fires spuriously and shows up as a teardown `ERROR` even though the test body passed. FRR starts near-empty on that baseline, so any test that brings BGP up/down (for example `ssh/test_ssh_stress`, which runs `config bgp startup/shutdown all`) grows `bgpd`/`zebra` far past the hardware-calibrated thresholds: ``` ERROR at teardown of test_ssh_stress[vlab-vpp-01] [ALARM]: frr_bgp:used memory usage increased by 250.0% ... (previous: 28.0 MB, current: 98.0 MB) [ALARM]: top:zebra memory usage increased by 195.9 MB ... (previous: 47.8 MB, current: 243.7 MB) ``` The plugin already downgrades these alarms to warnings on the **vs** virtual platform, and #23359 relaxed the `free` check on **vpp** for the same stated reason (*"vpp dataplane runs in host"*). This change completes that carve-out: `vpp` is a virtual/software platform whose process/FRR memory is not hardware-representative, so treat it like `vs` -- log memory alarms as warnings for visibility, but do not fail the test. Real hardware is unchanged (memory-leak gating is preserved where it is meaningful) and `vs` is unchanged. This also removes the need for per-test `disable_memory_utilization` skips on vpp (e.g. #25820). Fixes: stops the `memory_utilization` fixture from failing vpp tests at teardown on benign, non-hardware-representative memory growth, by treating vpp like vs (log memory alarms as warnings instead of failing).  ### Type of change  - [x] Bug fix - [x] Testbed and Framework(new/improvement) - [ ] New Test case - [ ] Skipped for non-supported platforms - [ ] Test case improvement ### Back port request  - [ ] 202311 - [ ] 202405 - [ ] 202411 - [ ] 202505 - [ ] 202511 - [ ] 202512 - [x] 202605 Tracking issue/work item for backport/cherry-pick request: Failure type: day-one issue ### Approach #### What is the motivation for this PR? The memory-leak check is valuable on real hardware, but on the virtual `vpp` baseline it produces false failures: memory footprints there are not hardware-representative (the dataplane runs in the host and FRR starts near-empty), so hardware-calibrated thresholds are tripped by benign, expected growth. Those false teardown errors show up as flaky nightly/PR failures across any vpp test that exercises BGP. #### How did you do it? At the single alarm choke point (`_handle_memory_threshold_exceeded`), extend the existing virtual-platform handling so `vpp` is treated the same as `vs`: log the alarm as a warning (for visibility) instead of storing an error that fails the test in the fixture teardown. This is one condition change (`asic_type in ("vs", "vpp")`) plus an explanatory comment; no thresholds and no other platform behavior change. Because the pre-commit `flake8` hook lints the whole touched file, it also surfaced 4 pre-existing `E231` false positives on f-string format specs (`f"{val:.1f}"`, a flake8-6.0.0/py3.12 quirk). These are suppressed with `# noqa: E231`, matching the suppression style already used in this file (e.g. `# noqa: E501`). No behavior change. #### How did you verify/test it? Elastictest, `ssh/test_ssh_stress.py` (which brings BGP up/down and trips the alarm) repeated 20x on the vpp KVM baseline (`vms-kvm-vpp-t1-lag`, t1-lag-vpp): - **Before (old code -- alarm fails the test):** https://elastictest.org/scheduler/testplan/6a47f6fda577b9de602ff366 -- 4/20 runs error at teardown, e.g. `[ALARM]: frr_bgp:used memory usage increased by 250.0% ... (28 -> 98 MB)`, where the `memory_utilization` fixture calls `pytest.fail` (the test body itself passes). - **After (this PR -- alarm is a warning):** https://elastictest.org/scheduler/testplan/6a4b43f5d38c80f6ad94be68 -- 20/20 pass; the same `frr_bgp`/`zebra` alarm is logged as a warning and no longer errors the test at teardown. Also: - `flake8 --max-line-length=120` clean (matches CI); `python -m py_compile` clean. #### Any platform specific information? Only the `vpp` asic path changes (its memory alarms become warnings). `vs` was already handled this way; physical-hardware platforms are unchanged and continue to fail on memory-threshold violations. #### Supported testbed topology if it's a new test case? N/A (framework plugin change, not a new test case). ### Documentation  No documentation changes. Signed-off-by: Augustine Lee <augustinelee@microsoft.com> Signed-off-by: mssonicbld <sonicbld@microsoft.com> Co-authored-by: augusdn <augustinelee@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions Bot requested review from ZhaohuiS, saiarcot895 and yejianquan July 1, 2026 16:19

saiarcot895 reviewed Jul 1, 2026

View reviewed changes

[pc/test_po_cleanup] Skip memory_utilization on vpp for test_po_clean…

c6d75f9

…up_after_reload Signed-off-by: Anant Kishor Sharma <anant.kishor-sharma@hpe.com>

github-actions Bot requested review from BYGX-wcr, xwjiang-ms and yutongzhang-microsoft July 1, 2026 16:48

saiarcot895 approved these changes Jul 2, 2026

View reviewed changes

yxieca approved these changes Jul 2, 2026

View reviewed changes

yxieca merged commit 2b76aa7 into sonic-net:master Jul 2, 2026
26 checks passed

yxieca mentioned this pull request Jul 2, 2026

[BMC][config_reload] Skip swss readiness gate in config_system_checks_passed #25027

Merged

8 tasks

augusdn mentioned this pull request Jul 6, 2026

[memory_utilization] Treat vpp like vs: warn (don't fail) on memory alarms #25911

Merged

12 tasks

mssonicbld mentioned this pull request Jul 23, 2026

[action] [PR:25911] [memory_utilization] Treat vpp like vs: warn (don't fail) on memory alarms #26510

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[pc/test_po_cleanup] Skip memory_utilization for test_po_cleanup_after_reload#25820

[pc/test_po_cleanup] Skip memory_utilization for test_po_cleanup_after_reload#25820
yxieca merged 2 commits into
sonic-net:masterfrom
AnantKishorSharma:vpp-po-cleanup-mem-util

AnantKishorSharma commented Jul 1, 2026 •

edited

Loading

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

saiarcot895 Jul 1, 2026

Uh oh!

AnantKishorSharma Jul 1, 2026

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

yxieca left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

AnantKishorSharma commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

saiarcot895 Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

AnantKishorSharma Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jul 1, 2026

Uh oh!

github-actions Bot commented Jul 1, 2026

Uh oh!

azure-pipelines Bot commented Jul 1, 2026

Uh oh!

yxieca left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AnantKishorSharma commented Jul 1, 2026 •

edited

Loading