Skip to content

Feature/ PFCWD Hardware Recovery HLD#2159

Open
pinky-nexthop wants to merge 15 commits into
sonic-net:masterfrom
nexthop-ai:feature/pfc-hardware-recovery-hld
Open

Feature/ PFCWD Hardware Recovery HLD#2159
pinky-nexthop wants to merge 15 commits into
sonic-net:masterfrom
nexthop-ai:feature/pfc-hardware-recovery-hld

Conversation

@pinky-nexthop

@pinky-nexthop pinky-nexthop commented Dec 16, 2025

Copy link
Copy Markdown

Summary

This PR introduces a comprehensive High Level Design document for PFC Watchdog Hardware Recovery feature that provides better accuracy in storm detection and restoration compared to software-based approaches.

Key Features

  • Hardware-based PFC recovery mechanism using SAI attributes with improved accuracy and reduced latency
  • Architecture refactoring with new PfcWdHwOrch orchestrator for clean separation of hardware/software logic
  • New CLI command: show pfcwd status with hardware-specific information including actual timer values and granularity
  • Flow diagrams illustrating hardware vs software recovery decision flows and detailed workflows
  • Comprehensive comparison between software and hardware recovery mechanisms

Hardware Recovery Advantages

  • Better accuracy in storm detection: Hardware operates at line rate without software polling delays
  • Better accuracy in storm restoration: More precise timing without dependency on software polling intervals
  • Reduced latency: Immediate hardware detection and recovery actions
  • Lower CPU overhead: Eliminates continuous software polling, reducing system load

Document Structure

  • Complete SONiC HLD format following established conventions
  • Placed in doc/pfcwd/ directory.
  • References existing PFC Watchdog Design and Test Plan documents
  • Includes SAI API specifications and comprehensive testing requirements

Technical Highlights

  • Hardware timer granularity constraints and actual vs configured values
  • Runtime platform capability detection for automatic hardware/software selection
  • Backward compatibility with existing software implementation
  • Error handling and validation for hardware constraints

Review Focus Areas

  • Architecture design and orchestrator refactoring approach
  • SAI attribute usage and hardware integration strategy
  • CLI design and user experience for hardware-specific information
  • Testing strategy and validation approach for accuracy improvements

Related Pull Requests

Index Description PR Number Pre-requisites
1 Add PFC Watchdog hardware state table definition sonic-swss-common#1169 None
2 Update th5 parameter to support hw pfcwd sonic-buildimage#26471 None
3 New cli "show pfcwd status" support sonic-utilities#4384 None
4 CLI UT for PFCWD status CLI sonic-utilities#4408 3
5 Refactor PFCWD sonic-swss#4358 None
6 Skeleton for PFCWD Hw Recovery sonic-swss#4403 1, 5
7 Implementation for Hw based PFCWD sonic-swss#4412 1, 5, 6
8 UT for flexcounters and orchagent changes sonic-swss#4418 1, 5, 6, 7
9 Add hardware-aware PFC watchdog test support sonic-mgmt#23430 None

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

- Comprehensive design document for hardware-based PFC recovery
- Includes architecture refactoring proposal with PfcWdHwOrch
- New CLI command: show pfcwd status with hardware-specific information
- SAI attributes and implementation details for hardware recovery
- Flow diagrams for hardware vs software recovery decision and workflow
- Testing requirements and validation approach
- Hardware timer granularity constraints and actual vs configured values
- Comparison table highlighting differences between software and hardware recovery

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Changed 'CLI/YANG' to 'CLI-YANG' to fix markdown anchor link
- Forward slash in header was breaking clickable navigation

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Updated TOC to reflect actual document structure
- Removed non-existent subsections 12.2, 12.3, 12.4
- Only section 12.1 exists in the streamlined testing section

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Better accuracy in storm detection and restoration timing
- Reduced latency compared to software polling-based approach
- Lower CPU overhead by eliminating continuous software polling
- Hardware operates at line rate without software delays

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Section 13 Open Points does not exist in the document
- Cleaned up Table of Contents to match actual document structure

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Added CLI Data Flow subsection to TOC for proper navigation
- Fixes parsing/navigation issue in section 5

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Simplified the display table description in mermaid flowchart
- Removed pipe characters that could cause parsing issues
- Improved readability of the CLI data flow diagram

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
- Created doc/pfcwd/ directory for PFC Watchdog documentation
- Moved pfc_hardware_recovery_hld.md from doc/qos/ to doc/pfcwd/
- Better organization following SONiC documentation structure
- Dedicated directory allows for future PFC Watchdog related documents

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
@pinky-nexthop
pinky-nexthop force-pushed the feature/pfc-hardware-recovery-hld branch from d793e08 to ee7c047 Compare December 16, 2025 11:41
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

- Added STATUS column to show pfcwd status command output to indicate
  success/failed programming status
- Updated CLI data flow diagram to include status validation
- Added example showing failed configuration due to unsupported timer ranges
- Updated hardware recovery workflow diagram to show event notification
  happening immediately after storm detection
- Added conditional logic for app_managed_recovery flag:
  - If true: wait for SAI_QUEUE_ATTR_PFC_DLR_INIT programming
  - If false: hardware automatically applies recovery action

Signed-off-by: Pinky Agrawal <pinky@nexthop.ai>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@pinky-nexthop pinky-nexthop changed the title Feature/pfc hardware recovery hld Feature/ PFCWD Hardware Recovery HLD Jan 13, 2026
Comment thread doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md
Comment thread doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md
No YANG model changes are required for this feature. The new `show pfcwd status` command displays operational state data, not configuration data.

**Rationale**: The new command displays runtime operational data (actual hardware values, granularity constraints, recovery type) rather than user configuration. This operational data is derived from:
- Platform hardware capabilities (granularity values)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any SAI capability query to get the hardware granularity? If so, please include SAI attributes

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the relevant attributes to the section

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

9 similar comments
@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

One or more co-authors of this pull request were not found. You must specify co-authors in commit message trailer via:

Co-authored-by: name <email>

Supported Co-authored-by: formats include:

  1. Anything <id+login@users.noreply.github.com> - it will locate your GitHub user by id part.
  2. Anything <login@users.noreply.github.com> - it will locate your GitHub user by login part.
  3. Anything <public-email> - it will locate your GitHub user by public-email part. Note that this email must be made public on Github.
  4. Anything <other-email> - it will locate your GitHub user by other-email part but only if that email was used before for any other CLA as a main commit author.
  5. login <any-valid-email> - it will locate your GitHub user by login part, note that login part must be at least 3 characters long.

Alternatively, if the co-author should not be included, remove the Co-authored-by: line from the commit message.

Please update your commit message(s) by doing git commit --amend and then git push [--force] and then request re-running CLA check via commenting on this pull request:

/easycla

@pinky-nexthop
pinky-nexthop force-pushed the feature/pfc-hardware-recovery-hld branch from 7dca809 to cda7331 Compare June 10, 2026 08:51
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

Drive the PfcWdHwOrch-vs-PfcWdSwOrch path selection solely from the SAI
capability query. Remove the SKU-based path determination and the operator
override for path selection, so bringing a platform onto hardware recovery
is purely a function of its SAI advertising the capability.

Addresses review feedback from @eddyk-nvidia, @kperumalbfn, @yunang-c.

Signed-off-by: pinky-nexthop <pinky@nexthop.ai>
@pinky-nexthop
pinky-nexthop force-pushed the feature/pfc-hardware-recovery-hld branch from cda7331 to 2ebf808 Compare June 10, 2026 08:56
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@vmittal-msft

Copy link
Copy Markdown
Contributor

@saksarav-nokia @ysmanman @arista-nwolfe please help review soon as we are planning to enable HW based PFCWD for upcoming SKUs and potentially for chassis as well.

@vmittal-msft

Copy link
Copy Markdown
Contributor

/easycla

@pinky-nexthop

Copy link
Copy Markdown
Author

@prsunny @vmittal-msft
Could you please help merge this to master?

@vmittal-msft

Copy link
Copy Markdown
Contributor

@pinky-nexthop i am waiting for @kperumalbfn to approve as well.

@github-project-automation github-project-automation Bot moved this from In Progress to Done in SONiC 202605 Release Jun 23, 2026
@vmittal-msft vmittal-msft reopened this Jun 24, 2026
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
No pipelines are associated with this pull request.

@vmittal-msft

Copy link
Copy Markdown
Contributor

@kperumalbfn @eddyk-nvidia please help approve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.