[vpp] SAIVPP unit-test harness (docker-sai-test-vpp) + devdocs by nicholasching · Pull Request #1950 · sonic-net/sonic-sairedis

nicholasching · 2026-06-23T16:04:13Z

Description of PR

Summary:
Adds the self-contained Docker harness (docker-sai-test-vpp) that runs the upstream OCP sai_test PTF suite against the real VPP SAI backend (libsaivs) in one container — VPP + saiserver + PTF + veth/AF_PACKET topology — and produces a per-test compatibility matrix. This is test scaffolding only (everything under .azure-pipelines/docker-sai-test-vpp/); it does not touch any production/runtime code, so it cannot affect existing sonic-sairedis functionality. Reviewers can start at .azure-pipelines/docker-sai-test-vpp/README.md and run_test.sh.

Fixes # (N/A — no upstream issue)

Type of change

Approach

What is the motivation for this PR?

There was no in-tree way to run the OCP sai_test suite against the VPP SAI backend and track per-test coverage. This harness provides that, plus the development docs and compatibility matrices used to drive the backend fixes.

How did you do it?

Added under .azure-pipelines/docker-sai-test-vpp/:

Dockerfile and run_test.sh (orchestrates Redis → veth/PortChannel topology → VPP → saiserver → PTF), plus T0 config (sai.profile, lanemap.ini, port-map.ini, ptf-port-map.ini) and gen_compatibility_matrix.py.
Config-signature grouping: because the VPP SAI backend builds the switch once per saiserver process, run_test.sh groups tests by their setUp config signature and restarts the backend per group, so one container can run any mix of tests.
ISOLATE_EACH_TEST (default on): run each test in its own group with a fresh backend + own config to eliminate cross-test config-reuse contamination; set 0 for faster grouped runs.
L3-over-LAG forwarding (harness portion): run_test.sh assigns the DUT-side connected IPs to each LAG (be<N>) and SVI (bvi<vlan>) so routed-to-LAG/SVI traffic forwards in the standalone PTF environment (the backend portion is in the backend-fixes PR).
Debug hold to keep VPP/saiserver/veths alive after a --debug run for vppctl inspection.
devdocs/, demodocs/, and results/ compatibility matrices; .gitignore for generated artifacts.

How did you verify/test it?

Running the harness against sai_route_test/sai_rif_test/sai_neighbor_test/sai_ecmp_test produces a compatibility matrix (current state: 34 PASS, captured in devdocs//results/). The harness is self-contained and does not build or run any production code.

Any platform specific information?

VPP platform test infrastructure only; isolated under .azure-pipelines/docker-sai-test-vpp/. No impact on the sonic-sairedis build or runtime behavior.

Documentation

This PR adds the harness README.md plus rolling/dated progress notes and deep-dive debug logs under .azure-pipelines/docker-sai-test-vpp/devdocs/ and demo docs under demodocs/.

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…test Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…dog, --relax Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…ature and restarting backend per group Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…ugging Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

UT-harness portion of the L3-over-LAG forwarding fix: run_test.sh assigns the DUT-side connected IPs to each LAG (be<N>) and SVI (bvi<vlan>) so routed-to-LAG /SVI traffic forwards in the standalone PTF environment. The backend portion (SwitchVppFdb.cpp) is in the VPP SAI backend PR. Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

linux-foundation-easycla · 2026-06-23T16:04:20Z

The committers listed above are authorized under a signed CLA.

✅ login: nicholasching / name: Nicholas Ching (01a0762, 02b11fa, 120e57f, 145aec2, 15776bb, 2661943, 4cd92d8, 4e1091b, 5780b35, 5cda18e, 77a77e9, 82ca412, b2ee2a3, b34a713, b882a30, c36a5f1, c78809e, dcc9a8f)

mssonicbld · 2026-06-23T16:04:21Z

/azp run

azure-pipelines · 2026-06-23T16:04:35Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-23T17:59:21Z

/azp run

azure-pipelines · 2026-06-23T17:59:33Z

Azure Pipelines successfully started running 1 pipeline(s).

…SER) - gen_compatibility_matrix.py: prefer defusedxml for parsing PTF JUnit XML (XXE-hardened), with a stdlib fallback; input is our own local test output. - Dockerfile: annotate the root entrypoint with a nosemgrep justification — the --privileged harness must run as root to start VPP/Redis/saiserver and manage veth/AF_PACKET; it is a disposable test container, never a deployed service. Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-06-23T18:04:03Z

/azp run

azure-pipelines · 2026-06-23T18:04:16Z

Azure Pipelines successfully started running 1 pipeline(s).

Satisfy tests/checkwhitespace.sh (Azure Build amd64 gate). Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…f using SAI teardown Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

…MULATE_SONIC Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-07-06T19:55:55Z

/azp run

azure-pipelines · 2026-07-06T19:57:28Z

Azure Pipelines successfully started running 1 pipeline(s).

nicholasching · 2026-07-06T20:11:01Z

/azp run

azure-pipelines · 2026-07-06T20:11:12Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-07-07T22:09:36Z

/azp run

azure-pipelines · 2026-07-07T22:09:48Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

mssonicbld · 2026-07-08T20:51:06Z

/azp run

azure-pipelines · 2026-07-08T20:51:20Z

Azure Pipelines successfully started running 1 pipeline(s).

dypet · 2026-07-21T16:21:10Z

+    # further packets remain - which is impossible for a real flood that delivers
+    # a copy to every member port. --relax makes verify_no_other_packets() a
+    # no-op, which is the intended/standard mode for flooding tests.
+    PTF_ARGS+=(--relax)


PTF_ARGS+=(--relax) is added unconditionally to all targets, not just the T0 cases mentioned above. Please scope --relax to only the flood test cases rather than the entire run.

Good catch, thank you @dypet. --relax disabled PTF's verify_no_other_packets() check, and applying it globally masked unexpected forwarding in non-flood tests. I changed the harness to add it only for an explicit allowlist of positive flood classes that use verify_each_packet_on_multiple_port_lists(). All other selectors now run with strict unexpected-packet verification. The change is in 67b5980b.

After more testing, we found this incorrect implementation to be hiding some background IPv6 lifecycle/control packets that strict PTF correctly treated as unexpected traffic, bringing down # of successful tests from 34 -> 19. The README.md at this PR stage has been updated to reflect that.

Upon implementing an opt-in, exact PTF filter for simulated-router Router Solicitation and MLDv2 startup frames, plus Linux IPv6 setup ordering that prevents DAD/router-discovery packets, the # of successful tests was restored 19 -> 30. These fixes involve both SAI and sonic-sairedis components. To prevent re-opening vpp_backend_fixes these fixes will be bundled with the others in the Phase 3 PRs. The associated SAI fix is here: opencomputeproject/SAI#2321

dypet · 2026-07-21T16:22:34Z

+    # Collect this invocation's JUnit XML into the shared results dir.
+    if [[ -n "$TEST_RESULTS_DIR" ]]; then
+        mkdir -p "$TEST_RESULTS_DIR"
+        cp -f "$xunit_dir"/*.xml "$TEST_RESULTS_DIR"/ 2>/dev/null || true


With ISOLATE_EACH_TEST=1 (default) or multi-group runs, two invocations that emit the same TEST-.xml filename will clobber each other, and gen_compatibility_matrix.py (which globs TEST-*.xml) would then undercount. Suggest namespacing the copied files per invocation (e.g. prefix with group/idx or test target) so no results are lost.

Thank you, I agree. I added deterministic invocation namespaces when collecting each report, for example TEST-group-3-test-8-sai_route_test.xml, while retaining the TEST- prefix expected by the matrix generator. The README also clears stale XML before a new run. This change is in 2ce71f7b. A collision test was performed, confirming both same-named source reports and testcase rows were retained.

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

mssonicbld · 2026-07-21T21:29:32Z

/azp run

azure-pipelines · 2026-07-21T21:29:45Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

mssonicbld · 2026-07-22T00:25:22Z

/azp run

azure-pipelines · 2026-07-22T00:25:34Z

Azure Pipelines: Successfully started running 1 pipeline(s).

nicholasching added 17 commits June 23, 2026 07:49

feat: configuration for 32 ports; phase 1, task 2

77a77e9

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

feat: configuration and run script; phase 1, task 3-4; fails to pass …

b2ee2a3

…test Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

fix: ifmap prefix, IPv6 disable, PortChannel emulation, veth-up watch…

145aec2

…dog, --relax Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

fix: run multiple tests sequentially by grouping tests by config sign…

4cd92d8

…ature and restarting backend per group Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

feat: script to generate compatabilty matrix from test results

2661943

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add feature development progress context

82ca412

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add build and usage instructions for UT harness

15776bb

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

chore: add hold when debug enabled to leave container running for deb…

dcc9a8f

…ugging Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add demo documentation

c36a5f1

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: add coverage by functionality

b882a30

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: minor updates to demo doc

01a0762

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: current dev state, compatibility matrix

c78809e

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: LAG-RIF neighbor/route + egress-disable progress (6-18)

4e1091b

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

feat: harness per-test isolation mode, restart backend between tests

5780b35

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: update to span full width

b34a713

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

docs: neighbor host-route local0 fix progress (6-19)

5cda18e

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

This was referenced Jun 23, 2026

[vpp] Dedicated binary-API socket and thread for asynchronous VPP events #1951

Open

[vpp] VPP SAI backend fixes: CPU queue, host-interface admin-up, L3-over-LAG/host-route forwarding, LAG egress-disable #1952

Open

nicholasching force-pushed the vpp_ut_harness branch from 73630af to 02b11fa Compare June 23, 2026 17:59

nicholasching force-pushed the vpp_ut_harness branch from 02b11fa to 4618cda Compare June 23, 2026 18:03

chore: strip trailing whitespace in progress-6-10 devdoc

7805b63

Satisfy tests/checkwhitespace.sh (Azure Build amd64 gate). Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

nicholasching added 2 commits July 6, 2026 12:54

docs: explain why the harness restarts the backend per test instead o…

4da0942

…f using SAI teardown Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

chore: drop sai_test overrides and drive PortChannel/RIF setup via SI…

e57cb6e

…MULATE_SONIC Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

nicholasching force-pushed the vpp_ut_harness branch from 641e9de to e57cb6e Compare July 6, 2026 19:55

nicholasching closed this Jul 7, 2026

nicholasching reopened this Jul 7, 2026

fix: satisfy aspell and SWSS_LOG_ENTER checks in UT harness preload shim

a01722c

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

This was referenced Jul 21, 2026

[Draft Only - Do Not Merge/Review] SAI VPP UT Harness - Phase 3 #1992

Open

SAI VPP UT Harness - PR Check Support in sonic-platform-vpp sonic-net/sonic-platform-vpp#264

Open

dypet requested changes Jul 21, 2026

View reviewed changes

nicholasching added 2 commits July 21, 2026 12:50

fix: scope relaxed packet checks to flood tests

67b5980

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

fix: preserve JUnit results across test invocations

2ce71f7

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Copilot AI review requested due to automatic review settings July 21, 2026 21:29

Copilot AI reviewed Jul 21, 2026

docs: record strict 19-test harness baseline

ae89be9

Signed-off-by: Nicholas Ching <nicholaslching@gmail.com>

Copilot AI review requested due to automatic review settings July 22, 2026 00:25

Copilot AI reviewed Jul 22, 2026

nicholasching requested a review from dypet July 22, 2026 01:06

dypet approved these changes Jul 22, 2026

View reviewed changes

Uh oh!

Conversation

nicholasching commented Jun 23, 2026

Description of PR

Type of change

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

Uh oh!

linux-foundation-easycla Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

mssonicbld commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

mssonicbld commented Jun 23, 2026

Uh oh!

azure-pipelines Bot commented Jun 23, 2026

Uh oh!

mssonicbld commented Jul 6, 2026

Uh oh!

azure-pipelines Bot commented Jul 6, 2026

Uh oh!

nicholasching commented Jul 6, 2026

Uh oh!

azure-pipelines Bot commented Jul 6, 2026

Uh oh!

mssonicbld commented Jul 7, 2026

Uh oh!

azure-pipelines Bot commented Jul 7, 2026

Uh oh!

mssonicbld commented Jul 8, 2026

Uh oh!

azure-pipelines Bot commented Jul 8, 2026

Uh oh!

dypet Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

nicholasching Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dypet Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

nicholasching Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jul 21, 2026

Uh oh!

azure-pipelines Bot commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jul 22, 2026

Uh oh!

azure-pipelines Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

linux-foundation-easycla Bot commented Jun 23, 2026 •

edited

Loading

nicholasching Jul 22, 2026 •

edited

Loading