[HA] swbus: replace stale peer connection on reconnect#209
Merged
vivekrnv merged 1 commit intoJul 22, 2026
Merged
Conversation
Signed-off-by: BYGX-wcr <wcr@live.cn>
|
/azp run |
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
Azure Pipelines: Successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Pull request overview
This PR fixes a SWBus failure mode after an HA peer reboot where a half-open/stale accepted (server-mode) connection can remain routable and cause messages to be queued locally but never delivered. The change ensures stale peer connections are replaced on reconnect and that equal next-hop identities refresh their stored send proxy so routing uses the latest live connection.
Changes:
- Replace older accepted server-mode connections for the same logical peer (by remote service path + connection type), unregistering stale routes and canceling the stale worker.
- Refresh the stored proxy for an equal next-hop identity by using
BTreeSet::replacerather thaninsertin route updates. - Add regression tests covering stale server replacement, preservation of independently owned client-mode connections, and proxy refresh for equal next-hops.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| crates/swbus-core/src/mux/multiplexer.rs | Refresh equal next-hop proxies via BTreeSet::replace and add a regression test ensuring the new proxy is used for routing. |
| crates/swbus-core/src/mux/conn.rs | Add a synchronous cancel() used by connection replacement to stop the worker after route cleanup. |
| crates/swbus-core/src/mux/conn_store.rs | Serialize connection replacement/registration, remove stale server-mode connections for the same peer, and add tests for stale replacement + client-path preservation. |
Comment on lines
89
to
+92
| pub fn conn_established(&self, conn: SwbusConn) { | ||
| // A single critical section prevents two reconnects from each preserving the other as stale. | ||
| let _guard = self.connection_update_lock.lock().unwrap(); | ||
| self.replace_stale_server_connections(conn.info()); |
zjswhhh
approved these changes
Jul 21, 2026
Contributor
|
LGTM |
vivekrnv
reviewed
Jul 21, 2026
|
|
||
| pub fn conn_established(&self, conn: SwbusConn) { | ||
| // A single critical section prevents two reconnects from each preserving the other as stale. | ||
| let _guard = self.connection_update_lock.lock().unwrap(); |
Contributor
There was a problem hiding this comment.
Just to confirm, locking is only during connection create and delete.
For sending any data on a live connection, this path is not hit right.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
After an HA peer NPU reboot, SWBus can retain a half-open incoming connection and continue routing HA messages through it even after the peer establishes a new connection. Messages appear locally queued but never reach the peer, which can leave HAMgrd stuck waiting for a vote reply and prevent publication of the
activate_rolepending operation.This change replaces older accepted connections when the same logical peer reconnects and refreshes the send proxy when an equal next-hop identity is registered again.
Reviewer entry points:
crates/swbus-core/src/mux/conn_store.rscrates/swbus-core/src/mux/multiplexer.rscrates/swbus-core/src/mux/conn.rsNo new dependencies are required.
Fixes sonic-net/sonic-buildimage#28481
Type of change
Approach
What is the motivation for this PR?
Server-side connection IDs include the peer ephemeral source port. After a peer reboot, the new connection therefore has a different ID while the old half-open connection can remain in the connection store and routing table until the system TCP timeout expires.
During that interval, SWBus can select the stale next hop. Queueing succeeds locally, so alternate routes are not attempted, but the peer never receives the message.
How did you do it?
swbs-toconnections.BTreeSet::replacewhen updating an equal next hop so the new send proxy replaces the stale proxy retained byinsert.How did you verify/test it?
Automated validation:
swbus-core: 36 tests passed, 0 failed.swbus-edge: 2 tests passed, 0 failed.Package and deployment validation:
dash-haanddash-ha-dbgsymDebian packages.dash-hadpu0throughdash-hadpu3on two NVIDIA SN4280 SmartSwitches.swbusdandhamgrdprocesses run the new payload.The repeated physical NPU reboot reproduction test for sonic-net/sonic-buildimage#28481 remains to be run.
Any platform specific information?
No. The connection identity and routing changes are generic SWBus behavior. Physical deployment validation was performed on NVIDIA SN4280 SmartSwitches.
Documentation
No documentation update is required. This change corrects internal connection lifecycle and route replacement behavior without changing configuration or public interfaces.