Skip to content

[TestFlexCounter] Use flag state rather than non-atomic counter for bulkChunksize test#1986

Open
justin-wong-ce wants to merge 1 commit into
sonic-net:masterfrom
justin-wong-ce:flakyfix
Open

[TestFlexCounter] Use flag state rather than non-atomic counter for bulkChunksize test#1986
justin-wong-ce wants to merge 1 commit into
sonic-net:masterfrom
justin-wong-ce:flakyfix

Conversation

@justin-wong-ce

Copy link
Copy Markdown
Contributor

Description of PR

Summary:
The existing initialCheckCount is not a reliable way to determine the state of the test-mocked configuration. This causes race conditions and makes the test flaky.
An example would be a poll occuring in between test iterations where the chunkSize is being changed.

Swapping to use a callback-based flag to explictly set when the configuration is fully applied and when it is not.
Fixes #1978

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation update
  • Test improvement

Approach

What is the motivation for this PR?

FlexConuter.bulkChunksize is flaky, causing many PR CIs to be flaky.

Work item tracking
  • Microsoft ADO (number only):

How did you do it?

Use a callback-flag mechanism to so that the test knows what to expect and what checks to run based on an explicitly known state.

How did you verify/test it?

Tests fully pass with ./tests --gtest_filter=FlexCounter.*-*addRemoveDashMeterCounter* --gtest_break_on_failure --gtest_repeat=100.

Excluding the addRemoveDashMeterCounter test since it is known to be flaky (#1985).

Any platform specific information?

Documentation

The existing `initialCheckCount` is not a reliable way to determine the
state of the test-mocked configuration. This causes many racey conditions
and makes the test flaky.
An example would be a poll occuring in between test iterations where the
`chunkSize` is being changed/

Swapping to use a callback-based flag to explictly set when the
configuration is fully applied and when it is not.

Signed-off-by: Justin Wong <jvwong@arista.com>
@justin-wong-ce justin-wong-ce changed the title Use flag state rather than non-atomic counter [TestFlexCounter.cpp] Use flag state rather than non-atomic counter[ Jul 10, 2026
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@justin-wong-ce justin-wong-ce changed the title [TestFlexCounter.cpp] Use flag state rather than non-atomic counter[ [TestFlexCounter] Use flag state rather than non-atomic counter[ Jul 10, 2026
@justin-wong-ce justin-wong-ce changed the title [TestFlexCounter] Use flag state rather than non-atomic counter[ [TestFlexCounter] Use flag state rather than non-atomic counter Jul 10, 2026
@justin-wong-ce justin-wong-ce changed the title [TestFlexCounter] Use flag state rather than non-atomic counter [TestFlexCounter] Use flag state rather than non-atomic counter for bulkChunksize test Jul 10, 2026
@justin-wong-ce

justin-wong-ce commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Unrelated CI fail:

[Check failure on line 13993 in Build log](https://github.com/sonic-net/sonic-sairedis/pull/1986/files#annotation_63835532184) 

@azure-pipelines
azure-pipelines
/ Azure.sonic-sairedis
Build log #L13993

Bash exited with code '137'.
 [Check failure on line 17 in Build log](https://github.com/sonic-net/sonic-sairedis/pull/1986/files#annotation_63835532198) 

@azure-pipelines
azure-pipelines
/ Azure.sonic-sairedis
Build log #L17

The process '/usr/bin/python' failed with exit code 1
 [Check failure on line 13993 in Build log](https://github.com/sonic-net/sonic-sairedis/pull/1986/files#annotation_63835532207) 

@azure-pipelines
azure-pipelines
/ Azure.sonic-sairedis
Build log #L13993

Bash exited with code '137'.
 [Check failure on line 10 in Build log](https://github.com/sonic-net/sonic-sairedis/pull/1986/files#annotation_63835532222) 

@azure-pipelines
azure-pipelines
/ Azure.sonic-sairedis
Build log #L10

ENOENT: no such file or directory, open '/__w/1/s/.diff-coverage/codecoverageProperties.json'

@justin-wong-ce

Copy link
Copy Markdown
Contributor Author

/azpw retry

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1161819:

✅Stage TestAsan:

  • Job vstest: retried.

✅Stage Test:

  • Job vstest: retried.
  • Job vstest: retried.

@justin-wong-ce

Copy link
Copy Markdown
Contributor Author

/azpw retry

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) jobs...

@mssonicbld

Copy link
Copy Markdown
Collaborator

Retrying failed(or canceled) stages in build 1161819:

✅Stage TestAsan:

  • Job vstest: retried.

✅Stage Test:

  • Job vstest: retried.
  • Job vstest: retried.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the FlexCounter.bulkChunksize unit test to reduce flakiness by replacing the initialCheckCount-based “initialization vs runtime” heuristic with an explicit phase signal driven by a callback from the shared testAddRemoveCounter helper.

Changes:

  • Add an optional onSetupComplete callback to testAddRemoveCounter and invoke it once the test’s FlexCounter/plugins/counters setup is complete.
  • Replace initialCheckCount logic in FlexCounter.bulkChunksize with a runtimeReady phase flag used by the mock_bulkGetStats implementation.
  • Adjust bulk chunk size assertions to align with the new phase gating.

uint32_t unifiedBulkChunkSize = 0;
int32_t initialCheckCount;
int32_t partialSupportingBulkObjectFactor;
bool runtimeReady = false;
Comment on lines 1578 to 1582
if (unifiedBulkChunkSize > 0)
{
if (object_count != unifiedBulkChunkSize
&& number_of_counters == allCounters.size()
&& object_count > 1)
{
EXPECT_EQ(object_count, unifiedBulkChunkSize);
}
continue;
}
// Skip re-probe polls (single-object capability probes
// that happen after the merge-back, with object_count==1).
// The per-counter assertions below check steady-state
// per-prefix chunk sizes, which only apply when
// object_count > 1. This matches the documented intent
// in the comment above and the unified-path guard.
if (object_count == 1)
{
EXPECT_EQ(object_count, unifiedBulkChunkSize);
continue;
}
// non zero unifiedBulkChunkSize indicates all counter IDs share the same bulk chunk size
uint32_t unifiedBulkChunkSize = 0;
int32_t initialCheckCount;
int32_t partialSupportingBulkObjectFactor;
@theasianpianist

Copy link
Copy Markdown
Contributor

@justin-wong-ce please review the copilot comments, esp the one about runtimeReady thread safety

@justin-wong-ce

Copy link
Copy Markdown
Contributor Author

@justin-wong-ce please review the copilot comments, esp the one about runtimeReady thread safety

Ack, will be looking at this soon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FlexCounter.bulkChunksize unit test is flaky

4 participants