Address racey SAI PHY Serdes object causing bad syncd setup by justin-wong-ce · Pull Request #1945 · sonic-net/sonic-sairedis

justin-wong-ce · 2026-06-18T01:10:08Z

Description of PR

Summary:
Mitigate a race condition in Port PHY Serdes initialization (syncd) where the SAI is still busy with the object from its creation when syncd's intialization attempts to read attributes from the object for syncd's own setup.

This race condition will cause a failing SAI call that is needed for initializing syncd, causing polls to the Port PHY Serdes object to fail continuously later on. The chances of this happening is around ~2.5% per config reload / reboot.

Do this by allowing 3 retries with 10ms wait intervals in between tries. The retry will only trigger when the failure reason is SAI_STATUS_OBJECT_IN_USE.

Fixes #1946

Type of change

Approach

What is the motivation for this PR?

Fix syslog ERRs that logs continuously when this scenario occurs.

Work item tracking

Microsoft ADO (number only):

How did you do it?

Added multiple attempts for interacing with Port PHY Serdes object over SAI during setup.

How did you verify/test it?

Continuous syslog ERRs no longer occur. The retry logic can be seen working in the syslogs as well. The most retry attempts observed is 1.

Any platform specific information?

Broadcom.

Documentation

Mitigate a race condition in Port PHY Serdes initialization (syncd) where the SAI is still busy with the object from its creation when syncd's intialization attempts to read attributes from the object for syncd's own setup. Do this by allowing 3 retries with 10ms wait intervals in between tries. The retry will only trigger when the failure reason is SAI_STATUS_OBJECT_IN_USE. Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T01:10:16Z

/azp run

azure-pipelines · 2026-06-18T01:10:27Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T02:00:49Z

/azp run

azure-pipelines · 2026-06-18T02:01:01Z

Azure Pipelines successfully started running 1 pipeline(s).

r12f · 2026-06-18T05:39:02Z

-            port_rid = attr.value.oid;
-            return true;
-        }
+            sai_status_t status = Base::m_vendorSai->get(Base::m_objectType, port_serdes_rid, 1, &attr);


hey Justin - the fix is a bit of "by-chance".

Also supposedly,

flex counter should not fail syncd init

flex counter should work after the objects gets created

so I wonder if we have looked deeper into the failure/race and see what might be happening?

As mentioned above, the race condition is from the SAI not being able to handle the Port PHY Serdes object creation and accessing of its attribute in extremely quick succession.

Either the SAI needs to be able to delay a request and not return SAI_STATUS_OBJECT_IN_USE, or Sonic (syncd initialization for this case) needs to be able to handle SAI_STATUS_OBJECT_IN_USE properly.

the fix is a bit of "by-chance"

I can change the fix such that it will retry until SAI does not return the SAI_STATUS_OBJECT_IN_USE, rather than just 3 times - this will remove the chance element.

I think it is reasonable to retry when the SAI returns SAI_STATUS_OBJECT_IN_USE since the object is busy and not available for read.

Please let me know what you think of the latest changes

what if we just simply skip the failed round and retry in the next pulling interval?

retrying 3 times is not going to guarantee the fix anyway.

This error is occuring during the initialization stage and this code that is being edited is for the initialization of Port PHY Serdes.
This will print the following ERR syslog once - during syncd's intialization:

30979 2026 Jun 7 20:56:03.499522 qspf201 ERR syncd#syncd: :- initAttrData: PORT_PHY_SERDES_ATTR: port_serdes_rid:0xb005700000000 has no serdes count attribute information 30980 2026 Jun 7 20:56:03.499522 qspf201 WARNING syncd#syncd: :- collectData: PORT_PHY_SERDES_ATTR: Failed to initialize attribute SAI_PORT_SERDES_ATTR_TX_FIR_TAPS_LIST for RID:0xb005700000000, skipping object

If SAI_STATUS_OBJECT_IN_USE is returned here and we don't retry, this will result missing info (maps) that is used to do other initalization steps later on, as it is treated as a failure.

If the info is missing, it will cause polling to fail all the time.
The following ERR syslog will print at every single poll interval:

26929 2026 Jun 7 20:55:59.007758 qspf201 ERR syncd#syncd: [none] SAI_API_PORT:brcm_sai_get_port_serdes_attribute_cmn:18248 Port 11 hardware lane count get failed with error -17. 26930 2026 Jun 7 20:55:59.007758 qspf201 ERR syncd#syncd: :- updatePortSerdesTapsCountMap: PORT_PHY_SERDES_ATTR: Failed to get port serdes count attr SAI_PORT_SERDES_ATTR_TX_FIR_COUNT for port_serdes RID:0xb005700000000, status:-17

@justin-wong-ce i see you have removed the retry loop, I dont think this is a right approach.
We should not retry without an upper bound. If we believe a retry of 3 times with a 10ms is not enough we can increase this to 5 retries but we should not leave it unbounded.
@r12f fyi.

@r12f please advise whether if we can go forward with 5 retries

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T18:29:47Z

/azp run

azure-pipelines · 2026-06-18T18:29:59Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T18:38:02Z

/azp run

azure-pipelines · 2026-06-18T18:38:15Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T18:43:58Z

/azp run

azure-pipelines · 2026-06-18T18:44:12Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-06-18T19:05:28Z

/azp run

azure-pipelines · 2026-06-18T19:05:43Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-18T21:04:02Z

/azp run

azure-pipelines · 2026-06-18T21:04:14Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2026-06-19T11:21:18Z

Hi, there are workflow run(s) waiting for approval, you may be first-time contributor. I will notify maintainers to help approve once PR is approved. Thanks!

_{---Powered by SONiC BuildBot}

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-07-07T20:53:33Z

/azp run

azure-pipelines · 2026-07-07T20:53:45Z

Azure Pipelines successfully started running 1 pipeline(s).

thisptr-sh · 2026-07-13T22:26:20Z

Hi @justin-wong-ce , can we get this PR merged? Are there any remaining blockers?

This issue is the biggest source of loganalyzer teardown failures across our SONiC 202511 hardware qualification on Broadcom TH5 (Arista 7060X6-512). We're seeing the same periodic ERR from initAttrData (port_serdes_rid:0x1005700000000 has no lane count information) every 10 second.

justin-wong-ce · 2026-07-13T22:52:03Z

Hi @justin-wong-ce , can we get this PR merged? Are there any remaining blockers?

This issue is the biggest source of loganalyzer teardown failures across our SONiC 202511 hardware qualification on Broadcom TH5 (Arista 7060X6-512). We're seeing the same periodic ERR from initAttrData (port_serdes_rid:0x1005700000000 has no lane count information) every 10 second.

I am awaiting feedback in this thread: #1945 (comment)

I do not have merge permissions. To pre-emptively resolve any potential code coerage issues I am working on a unit test at the moment.

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-07-14T00:15:09Z

/azp run

azure-pipelines · 2026-07-14T00:15:20Z

Azure Pipelines successfully started running 1 pipeline(s).

justin-wong-ce · 2026-07-14T00:45:49Z

Note: I am aware of the cherry-pick conflict to 202511. I have a local branch that addresses this conflict ready and will make a direct backport when this PR is approved.

    flexCounter->removeCounter(testPortSerdesOid);
 }

+TEST_F(TestPortPhySerdesAttr, RetryOnObjectInUseThenSucceed)


Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-07-14T01:17:39Z

/azp run

azure-pipelines · 2026-07-14T01:17:50Z

Azure Pipelines successfully started running 1 pipeline(s).

tjchadaga · 2026-07-14T08:10:08Z

@justin-wong-ce - are these failure logs only seen on a specific platform?

justin-wong-ce · 2026-07-14T16:40:34Z

@justin-wong-ce - are these failure logs only seen on a specific platform?

I do not have access to other platforms - I can only test on Broadcom.

I cannot say for sure if it happens on other platforms so I only noted Broadcom in the PR description.

mssonicbld · 2026-07-14T16:47:24Z

/azp run

azure-pipelines · 2026-07-14T16:47:36Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld · 2026-07-14T17:11:53Z

/azp run

azure-pipelines · 2026-07-14T17:12:06Z

Azure Pipelines: Successfully started running 1 pipeline(s).

justin-wong-ce · 2026-07-14T19:59:03Z

Restarting CI due to failure from:

The process '/usr/bin/python' failed with exit code 1
BuildTrixie • amd64 • [Auto] Generate build coverage properties

ENOENT: no such file or directory, open '/__w/1/s/.diff-coverage/codecoverageProperties.json'
BuildTrixie • amd64 • [Auto] Update build coverage properties
Bash exited with code '100'.
Test • vstest • Install dependencies

justin-wong-ce · 2026-07-14T19:59:08Z

/azpw retry

mssonicbld · 2026-07-14T19:59:10Z

Retrying failed(or canceled) jobs...

mssonicbld · 2026-07-14T19:59:12Z

Retrying failed(or canceled) stages in build 1164893:

✅Stage Test:

Job vstest: retried.

r12f · 2026-07-15T22:24:48Z

hi @tjchadaga - do you mind to help check this PR?

retriving -> retrieving

d67380d

Signed-off-by: Justin Wong <jvwong@arista.com>

StormLiangMS requested a review from lolyu June 18, 2026 05:33

davidm-arista added Request for 202511 Branch Request for 202605 Branch labels Jun 18, 2026

StormLiangMS requested review from r12f and removed request for r12f June 18, 2026 05:35

r12f reviewed Jun 18, 2026

View reviewed changes

github-advanced-security AI found potential problems Jun 18, 2026

View reviewed changes

Comment thread syncd/FlexCounter.cpp Fixed

Comment thread syncd/FlexCounter.cpp Fixed

Change to more deterministic handling of SAI_STATUS_OBJECT_IN_USE

0d31fcf

Signed-off-by: Justin Wong <jvwong@arista.com>

Cast revised handling to other code path

117aa72

Signed-off-by: Justin Wong <jvwong@arista.com>

Further casts

b2eac07

Signed-off-by: Justin Wong <jvwong@arista.com>

Fix whiteline and typo

220a847

Signed-off-by: Justin Wong <jvwong@arista.com>

justin-wong-ce closed this Jun 18, 2026

justin-wong-ce reopened this Jun 18, 2026

bump tries count, make code more easy to understand

faf8c21

Signed-off-by: Justin Wong <jvwong@arista.com>

Add unit test for retry logic

27efe15

Signed-off-by: Justin Wong <jvwong@arista.com>

mssonicbld added the Cherry Pick Conflict_202511 label Jul 14, 2026

github-advanced-security AI found potential problems Jul 14, 2026

View reviewed changes

Comment thread unittest/syncd/TestPortPhySerdesAttr.cpp

flexCounter->removeCounter(testPortSerdesOid);

}

TEST_F(TestPortPhySerdesAttr, RetryOnObjectInUseThenSucceed)

added comments for new test cases

a33c0e8

Signed-off-by: Justin Wong <jvwong@arista.com>

justin-wong-ce force-pushed the port-phy-retry branch from 3ac7f55 to a33c0e8 Compare July 14, 2026 01:06

tjchadaga reviewed Jul 14, 2026

View reviewed changes

Comment thread syncd/FlexCounter.cpp Outdated

tjchadaga requested a review from prgeor July 14, 2026 08:03

fix typo

8aa4ae9

Signed-off-by: Justin Wong <jvwong@arista.com>

justin-wong-ce force-pushed the port-phy-retry branch from 23718ac to 8aa4ae9 Compare July 14, 2026 17:10

Uh oh!

Conversation

justin-wong-ce commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Approach

What is the motivation for this PR?

Work item tracking

How did you do it?

How did you verify/test it?

Any platform specific information?

Documentation

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

r12f Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

r12f Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r12f Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

dhanasekar-arista Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

justin-wong-ce Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 18, 2026

Uh oh!

azure-pipelines Bot commented Jun 18, 2026

Uh oh!

mssonicbld commented Jun 19, 2026

Uh oh!

mssonicbld commented Jul 7, 2026

Uh oh!

azure-pipelines Bot commented Jul 7, 2026

Uh oh!

thisptr-sh commented Jul 13, 2026

Uh oh!

justin-wong-ce commented Jul 13, 2026

Uh oh!

mssonicbld commented Jul 14, 2026

justin-wong-ce commented Jun 18, 2026 •

edited

Loading

r12f Jun 18, 2026 •

edited

Loading