Skip to content

[NPU-driven HA] Ack standalone before peer activates dead role#203

Merged
BYGX-wcr merged 2 commits into
sonic-net:masterfrom
BYGX-wcr:fix-active-ack-standalone-before-peer-dead
Jul 6, 2026
Merged

[NPU-driven HA] Ack standalone before peer activates dead role#203
BYGX-wcr merged 2 commits into
sonic-net:masterfrom
BYGX-wcr:fix-active-ack-standalone-before-peer-dead

Conversation

@BYGX-wcr

@BYGX-wcr BYGX-wcr commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Description of PR

Summary:
During a planned shutdown of a standby HA scope, the active side previously gated its transition from Active to Standalone on the standby peer entering the Dead role. This ordering is incorrect: the active must ack the standalone role first so that it owns all traffic before the standby activates its dead role. Otherwise there is a window where the active is still in SwitchingToStandalone while the standby has already dropped out, leaving traffic unowned / inline sync unacked.

This PR flips the ordering so the active acks standalone first, and the standby only activates its dead role after observing the active's standalone ack.

Fixes # (N/A)

Type of change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation update
  • Test improvement

Approach

What is the motivation for this PR?

Correct the NPU-driven HA planned-shutdown state-transition ordering to avoid a traffic-ownership gap between the active and standby HA scopes.

How did you do it?

  • Active side (apply_pending_state_side_effects, SwitchingToStandalone): on PeerShutdownRequested, enter Standalone immediately instead of waiting for the peer to become Dead.
  • Standby side (next_state): gate the transition to Destroying (dead-role activation) on the active peer having acked the standalone role — both in the forced Shutdown path and via a new Standby branch that fires when the peer's acked ASIC role becomes standalone during a planned shutdown (desired_ha_state == Dead).

How did you verify/test it?

  • Updated the affected NPU-driven unit tests (ha_scope_npu_launch_to_standby_then_down, ha_scope_npu_active_to_standalone_on_peer_shutdown_and_peer_rejoin, ha_scope_npu_standby_planned_shutdown_and_relaunch_to_standby) to reflect the new ordering.
  • cargo test -p hamgrd --bin hamgrd actors::ha_scope — all 13 tests pass.
  • cargo fmt -- --check clean; cargo clippy -p hamgrd reports no new warnings.

Any platform specific information?

NPU-driven HA only. DPU-driven mode is unaffected.

Documentation

N/A — behavior fix aligned with the SmartSwitch HA HLD planned-shutdown flow.

During a planned shutdown, the active side previously gated its transition
to Standalone on the standby peer entering the Dead role. This ordering was
incorrect: the active must ack the standalone role first so it owns all
traffic before the standby activates its dead role.

- Active side: on PeerShutdownRequested, enter Standalone immediately instead
  of waiting for the peer to become Dead.
- Standby side: gate the transition to Destroying (dead role activation) on
  the active peer having acked the standalone role.

Update the affected NPU-driven unit tests to reflect the new ordering.

Signed-off-by: BYGX-wcr <wcr@live.cn>
Copilot AI review requested due to automatic review settings July 3, 2026 05:05
@mssonicbld

Copy link
Copy Markdown

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes the NPU-driven planned-shutdown ordering in hamgrd HA scope logic so the active side acknowledges Standalone first, and the standby side only activates Dead after observing that standalone ack, eliminating a traffic-ownership / inline-sync-ack gap during standby shutdown.

Changes:

  • Updated SwitchingToStandalone side effects on the active side to enter Standalone immediately upon PeerShutdownRequested (planned shutdown).
  • Updated standby-side state transitions to gate Dead activation (Destroying) on the peer’s acked ASIC role being standalone, including a new Standby handling path.
  • Updated NPU-driven unit tests to reflect and enforce the new ordering.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
crates/hamgrd/src/actors/ha_scope/npu.rs Changes the planned-shutdown transition sequencing: active acks Standalone first; standby waits for that ack before activating Dead.
crates/hamgrd/src/actors/ha_scope/mod.rs Updates tests to assert the new “standalone ack before dead activation” ordering during planned shutdown flows.

@mssonicbld

Copy link
Copy Markdown

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@BYGX-wcr
BYGX-wcr merged commit 8243bbc into sonic-net:master Jul 6, 2026
4 checks passed
@BYGX-wcr
BYGX-wcr deleted the fix-active-ack-standalone-before-peer-dead branch July 6, 2026 20:31
@mssonicbld

Copy link
Copy Markdown

Cherry-pick PR to 202605: #204

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants