From eda117f7ae3821a0c189b6e3ac1ae4ed893dc0b6 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:18:26 +0000 Subject: [PATCH 01/15] Add PFC Watchdog Hardware Recovery HLD - Comprehensive design document for hardware-based PFC recovery - Includes architecture refactoring proposal with PfcWdHwOrch - New CLI command: show pfcwd status with hardware-specific information - SAI attributes and implementation details for hardware recovery - Flow diagrams for hardware vs software recovery decision and workflow - Testing requirements and validation approach - Hardware timer granularity constraints and actual vs configured values - Comparison table highlighting differences between software and hardware recovery Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 345 +++++++++++++++++++++++++++ 1 file changed, 345 insertions(+) create mode 100644 doc/qos/pfc_hardware_recovery_hld.md diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md new file mode 100644 index 00000000000..d5dedf6aeca --- /dev/null +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -0,0 +1,345 @@ +# PFC Watchdog Hardware Recovery + +## Table of Contents + +- [Revision](#revision) +- [Scope](#scope) +- [Abbreviations](#abbreviations) +- [1. Overview](#1-overview) +- [2. Requirements](#2-requirements) +- [3. Architecture Design](#3-architecture-design) +- [4. High-Level Design](#4-high-level-design) + - [4.1 Hardware Recovery Mechanism](#41-hardware-recovery-mechanism) + - [4.2 PFC Watchdog Orchagent Refactoring](#42-pfc-watchdog-orchagent-refactoring) +- [5. CLI Changes](#5-cli-changes) +- [6. SAI API](#6-sai-api) + - [6.1 SAI Attributes](#61-sai-attributes) + - [6.2 SAI Events](#62-sai-events) + - [6.3 SAI Statistics](#63-sai-statistics) +- [7. Configuration and Management](#7-configuration-and-management) +- [8. Manifest](#8-manifest) +- [9. CLI/YANG Model Enhancements](#9-cliyangmodel-enhancements) +- [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) +- [11. Restrictions/Limitations](#11-restrictionslimitations) +- [12. Testing Requirements/Design](#12-testing-requirementsdesign) + - [12.1 Unit Test Cases](#121-unit-test-cases) + - [12.2 System Test Cases](#122-system-test-cases) + - [12.3 Performance Testing](#123-performance-testing) + - [12.4 Compatibility Testing](#124-compatibility-testing) +- [13. Open Points](#13-open-points) + +## Revision + +| Rev | Date | Author | Change Description | +|-----|------------|---------------|--------------------| +| 0.1 | 12/13/2024 | PFC Team | Initial version | + +## Scope + +This document describes the design for hardware-based PFC (Priority Flow Control) watchdog recovery mechanism in SONiC. It covers the extension of existing software-based PFC watchdog functionality to support hardware-accelerated detection and recovery. + +## Abbreviations + +| Abbreviation | Description | +|--------------|-------------| +| PFC | Priority Flow Control | +| HLD | High Level Design | +| SAI | Switch Abstraction Interface | +| CLI | Command Line Interface | +| ACL | Access Control List | +| TC | Traffic Class | +| DLR | Deadlock Recovery | +| DLD | Deadlock Detection | + +## 1. Overview + +This design document is an extension of the existing PFC watchdog functionality described in the [PFC Watchdog Design](https://github.com/sonic-net/SONiC/wiki/PFC-Watchdog-Design). Newer network chips are capable of PFC storm detection and mitigation in the hardware without software intervention. Current implementation of software based PFC watchdog detection depends on constant polling of the queue's pause state, mitigation requires installation of ACLs to take drop action and restoration requires constant polling of PFC frames received on the queue. + +This document provides an overview of the programming requirements to enable hardware based recovery for PFC watchdog. + +## 2. Requirements + +The hardware-based PFC watchdog recovery feature shall provide: + +1. **Hardware Detection**: Automatic PFC deadlock detection in hardware without software polling +2. **Hardware Recovery**: Automatic recovery actions (drop/forward) performed by hardware +3. **Backward Compatibility**: Existing software-based PFC watchdog functionality remains unchanged +4. **Runtime Selection**: Automatic selection between hardware and software implementations based on platform capabilities +5. **Enhanced CLI**: New `show pfcwd status` command to display recovery type, actual hardware timer values, and granularity information +6. **Error Handling**: Proper validation and error reporting for hardware timer granularity constraints + +## 3. Architecture Design + +The architecture maintains separation between hardware and software PFC watchdog implementations while sharing common CLI parsing and validation logic. + +``` +Current: Orch -> PfcWdOrch -> PfcWdSwOrch +Proposed: Orch -> PfcWdBaseOrch -> {PfcWdHwOrch | PfcWdSwOrch} +``` + +## 4. High-Level Design + +### 4.1 Hardware Recovery Mechanism + +Hardware recovery mechanism requires the application to program certain SAI attributes to initiate hardware based PFC deadlock detection and recovery. The below table highlights the key differences between hardware and software recovery. +Each phase of hardware detection and recovery happens based on the programming of a corresponding SAI attribute. + +| Phase | Software Recovery | Hardware Recovery | +| ----- | ----- | ----- | +| Deadlock detection requirement | Polling period for lua script configured in S/W | Enable h/w deadlock detection and recovery. Configure detection timer. Register callback for h/w events | +| Detection mechanism | Lua script polls the queue to check if queue received PFC frames continuously for detection time | H/w detects the deadlock and publishes an event to the s/w to take action, if required. | +| PFC pause frame handling | Software disables PFC by clearing the PFC mask bit for the affected queue (ignores incoming PFC pause frames) | Hardware sets the registers that ignores PFC pause frames for the affected queue during storm mitigation | +| Mitigation | Orchagent programs ACLS to take drop action or programs a zero-buffer profile | H/w takes the drop/forward action | +| Recovery requirement | Polling period for lua script configured in S/W | Configure recovery timer attribute | +| Recovery | Lua script polls if the queue remains unpaused for restoration-time amount of time to recover | H/w recovers the queue and publishes an event to the s/w to take action, if required. | + +**Timer granularity constraints:** +- Hardware may have granularity limitations (e.g., 100ms granularity) +- Configured values may differ from actual hardware-programmed values +- Upper limits may apply based on platform (e.g., Broadcom: 1-15 × granularity) + +#### 4.1.1 Hardware vs Software Recovery Flow + +The following diagram illustrates the decision flow for selecting hardware vs software recovery: + +```mermaid +flowchart TD + A[PFC Watchdog Configuration] --> B{Platform Supports
Hardware Recovery?} + B -->|Yes| C[Initialize PfcWdHwOrch] + B -->|No| D[Initialize PfcWdOrch
Software Recovery] + + C --> E[Configure SAI Hardware
Watchdog Attributes] + E --> F[Hardware Detection
& Recovery Active] + + D --> G[Configure Lua Script
Polling & Handlers] + G --> H[Software Detection
& Recovery Active] + + F --> I[PFC Storm Detected
by Hardware] + H --> J[PFC Storm Detected
by Lua Script] + + I --> K[Hardware Automatic
Recovery Action] + J --> L[Software Handler
Recovery Action] + + K --> M[Hardware Event
Notification to SW] + L --> N[Update Counters
& State] + + M --> P[Software Counter
Polling & State Update] + N --> O[Update Statistics
& CLI Display] + P --> O +``` + +#### 4.1.2 Hardware Recovery Workflow + +The following diagram shows the detailed hardware recovery process: + +```mermaid +sequenceDiagram + participant CLI as CLI/Config + participant Orch as PfcWdHwOrch + participant SAI as SAI Layer + participant HW as Hardware + participant DB as Counters DB + + CLI->>Orch: Configure PFC Watchdog + Orch->>SAI: Set Detection Timer + Orch->>SAI: Set Restoration Timer + Orch->>SAI: Set Recovery Action + Orch->>SAI: Enable Hardware Watchdog + SAI->>HW: Program Hardware Timers + + Note over HW: PFC Storm Occurs + HW->>HW: Detect Storm (Hardware Timer) + HW->>HW: Apply Recovery Action + HW->>SAI: Generate Event Notification + SAI->>Orch: Storm Detected Event + Orch->>DB: Update Storm Counters + + Note over HW: Storm Subsides + HW->>HW: Restoration Timer Expires + HW->>HW: Restore Normal Operation + HW->>SAI: Generate Restoration Event + SAI->>Orch: Storm Restored Event + Orch->>DB: Update Restoration Counters +``` + +### 4.2 PFC Watchdog Orchagent Refactoring + +This document outlines a proposal to refactor the PFC watchdog orchestrator architecture to support both hardware-based and software-based PFC recovery mechanisms. + +#### Design Goals + +- **Clean separation**: Hardware-specific logic isolated from software watchdog logic +- **Backward compatibility**: Existing software implementation remains unchanged +- **Code reuse**: Common functionality (CLI parsing, validation) remains in one place for both. + +#### Current Architecture + +``` +Orch (base class) + └── PfcWdOrch (template class) + └── PfcWdSwOrch (template class) +``` + +#### Proposed Architecture + +``` +Orch (base class) + └── PfcWdBaseOrch (new base class) + ├── PfcWdHwOrch (new hardware-based class - no handlers) + └── PfcWdSwOrch (existing template class) +``` + +#### Implementation Details + +`PfcWdBaseOrch` is the new base class that will have the common logic of parsing the CLI and creating the tasks. This implementation exists in `PfcWdOrch` today and will be moved to this new class. `PfcWdOrch` will be removed and renamed to `PfcWdBaseOrch` without the template parameters. + +`PfcWdHwOrch` is the new class that will handle hardware recovery mechanisms. There will be no handlers defined in hardware-based recovery unlike software recovery which defines drop and forward handlers. To update the counters periodically, `PfcWdHwOrch` itself will provide a function to read the required SAI attributes and get updated counters. + +Runtime selection in `orchdaemon.cpp`: + +```c +// Check hardware capability +bool pfcHwRecoverySupported = gSwitchOrch->checkPfcHwRecoverySupport(); + +if (pfcHwRecoverySupported) +{ + SWSS_LOG_NOTICE("Starting hardware-based PFC watchdog (no handlers)"); + m_orchList.push_back(new PfcWdHwOrch( + m_configDb, + pfc_wd_tables, + portStatIds, + queueStatIds, + queueAttrIds)); +} +``` + + + + +## 5. CLI Changes + +All existing CLI commands for PFC watchdog will remain unchanged as documented in [section 2.7 of the PFC Watchdog Design](https://github.com/sonic-net/SONiC/wiki/PFC-Watchdog-Design#27-cli). + +Hardware recovery might have some constraints as listed below: + +**Timer granularity** +There may be a granularity associated with the timers, so it is possible that the value configured in h/w is different from the value configured through the cli. +For example: if the hardware granularity is 100 ms and configured detection time through cli is 250 ms, then hardware might program either 200/300 ms. + +**Upper limit on the timer value** +Some platforms like broadcom only allow the timer value to be within the range <1-15> multiplied with the current granularity. So, it is possible that the value in the cli config cannot be programmed in the hardware. There is no mechanism to notify the config status today. + +**New CLI command** +We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. + +#### 5.1 CLI Data Flow + +The following diagram illustrates how the new `show pfcwd status` command retrieves and displays information: + +```mermaid +flowchart LR + A[show pfcwd status] --> B{Recovery Type?} + + B -->|Hardware| C[Query SAI Attributes] + B -->|Software| D[Return N/A Values] + + C --> E[Get Actual HW Timer Values] + C --> F[Get Platform Granularity] + C --> G[Get Recovery Type] + + E --> H[Format Display Data] + F --> H + G --> H + D --> H + + H --> I[Display Table:
PORT | RECOVERY TYPE | HW DETECTION TIME |
DETECTION GRANULARITY | HW RESTORATION TIME |
RESTORATION GRANULARITY] +``` + +```shell +admin@sonic:~$ show pfcwd status +PORT RECOVERY TYPE HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY +---------- ------------- ------------------- --------------------- --------------------- ----------------------- +Ethernet0 hardware 300 100ms 500 100ms +Ethernet4 software N/A N/A N/A N/A +Ethernet8 hardware 200 50ms 400 100ms +``` + +## 6. SAI API + +Following SAI statistics and attributes are used in this feature: + +### 6.1 SAI Attributes + +- `SAI_QUEUE_ATTR_ENABLE_PFC_DLDR` - Enable PFC deadlock detection and recovery +- `SAI_SWITCH_ATTR_QUEUE_PFC_DEADLOCK_NOTIFY` - Register callback for PFC deadlock events +- `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` - Detection timer intervals per port/per TC +- `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` - Recovery timer intervals per port/per TC +- `SAI_SWITCH_ATTR_PFC_DLR_PACKET_ACTION` - Configure drop/forward action +- `SAI_QUEUE_ATTR_PFC_DLR_INIT` - Manual recovery control + +### 6.2 SAI Events + +- `SAI_QUEUE_PFC_DEADLOCK_EVENT_TYPE_DETECTED` - PFC deadlock detected event +- `SAI_QUEUE_PFC_DEADLOCK_EVENT_TYPE_RECOVERED` - PFC deadlock recovered event + +### 6.3 SAI Statistics + +Hardware-based PFC watchdog will use existing SAI statistics for counter collection: + +- `SAI_QUEUE_STAT_PACKETS` - Queue packet counters +- `SAI_QUEUE_STAT_BYTES` - Queue byte counters +- `SAI_QUEUE_STAT_DROPPED_PACKETS` - Queue dropped packet counters +- `SAI_QUEUE_STAT_DROPPED_BYTES` - Queue dropped byte counters + +## 7. Configuration and Management + +The hardware-based PFC watchdog uses the same configuration interface as the existing software implementation: + +- Configuration is done through CONFIG_DB PFC_WD table +- Runtime selection between hardware and software is automatic based on platform capabilities +- No additional configuration parameters are required for hardware mode + +## 8. Manifest + +Not applicable - this is a core SONiC feature enhancement. + +## 9. CLI/YANG Model Enhancements + +No YANG model changes are required for this feature. The new `show pfcwd status` command displays operational state data, not configuration data. + +**Rationale**: The new command displays runtime operational data (actual hardware values, granularity constraints, recovery type) rather than user configuration. This operational data is derived from: +- Platform hardware capabilities (granularity values) +- Runtime hardware programming (actual timer values) +- Platform detection logic (recovery type selection) + +The existing PFC watchdog configuration YANG model remains unchanged as all user-configurable parameters (detection_time, restoration_time, action, etc.) are already supported. + +## 10. Warmboot and Fastboot Design Impact + +There are no impacts to warmboot or fastboot. The hardware PFC watchdog configuration will be restored during the warmboot process through the standard CONFIG_DB restoration mechanism. + +## 11. Restrictions/Limitations + +- Hardware timer granularity constraints may prevent exact timer value programming +- Platform-specific upper limits on timer values may apply +- Manual recovery control requires platform support for `SAI_QUEUE_ATTR_PFC_DLR_INIT` + +## 12. Testing Requirements/Design + +All existing PFC watchdog testing remains unchanged as documented in the [PFC Watchdog Test Plan](https://github.com/sonic-net/SONiC/wiki/PFC-Watchdog-Test-Plan). + +### 12.1 Additional Test Cases for Hardware Recovery + +#### New CLI Command Testing +- Verify `show pfcwd status` command displays all required fields correctly +- Test recovery type display (hardware vs software) +- Validate hardware timer values and granularity information display +- Test N/A display for software recovery ports +- Verify column alignment and formatting +- Test command behavior on mixed hardware/software port configurations + +#### Hardware Recovery Functionality Testing +- Verify hardware capability detection and automatic selection between hardware/software implementations +- Test SAI attribute programming for hardware watchdog configuration +- Validate timer granularity constraints and hardware programming +- Test error handling for unsupported timer values and hardware programming failures \ No newline at end of file From daf7336308becc8a73b7b3e88c22005a8b23adce Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:21:22 +0000 Subject: [PATCH 02/15] Fix section 9 anchor link in Table of Contents - Changed 'CLI/YANG' to 'CLI-YANG' to fix markdown anchor link - Forward slash in header was breaking clickable navigation Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md index d5dedf6aeca..2a1411b3e36 100644 --- a/doc/qos/pfc_hardware_recovery_hld.md +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -18,7 +18,7 @@ - [6.3 SAI Statistics](#63-sai-statistics) - [7. Configuration and Management](#7-configuration-and-management) - [8. Manifest](#8-manifest) -- [9. CLI/YANG Model Enhancements](#9-cliyangmodel-enhancements) +- [9. CLI-YANG Model Enhancements](#9-cli-yang-model-enhancements) - [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) - [11. Restrictions/Limitations](#11-restrictionslimitations) - [12. Testing Requirements/Design](#12-testing-requirementsdesign) @@ -303,7 +303,7 @@ The hardware-based PFC watchdog uses the same configuration interface as the exi Not applicable - this is a core SONiC feature enhancement. -## 9. CLI/YANG Model Enhancements +## 9. CLI-YANG Model Enhancements No YANG model changes are required for this feature. The new `show pfcwd status` command displays operational state data, not configuration data. From fcf84a69ed56c72af8a5012d4e5418759b9cb6a8 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:22:43 +0000 Subject: [PATCH 03/15] Fix section 12 Table of Contents entries - Updated TOC to reflect actual document structure - Removed non-existent subsections 12.2, 12.3, 12.4 - Only section 12.1 exists in the streamlined testing section Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md index 2a1411b3e36..c4561a7f758 100644 --- a/doc/qos/pfc_hardware_recovery_hld.md +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -22,10 +22,7 @@ - [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) - [11. Restrictions/Limitations](#11-restrictionslimitations) - [12. Testing Requirements/Design](#12-testing-requirementsdesign) - - [12.1 Unit Test Cases](#121-unit-test-cases) - - [12.2 System Test Cases](#122-system-test-cases) - - [12.3 Performance Testing](#123-performance-testing) - - [12.4 Compatibility Testing](#124-compatibility-testing) + - [12.1 Additional Test Cases for Hardware Recovery](#121-additional-test-cases-for-hardware-recovery) - [13. Open Points](#13-open-points) ## Revision From b88c152e4a4679dacec65d17b44cc2ea6b00f636 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:25:51 +0000 Subject: [PATCH 04/15] Add hardware recovery accuracy benefits - Better accuracy in storm detection and restoration timing - Reduced latency compared to software polling-based approach - Lower CPU overhead by eliminating continuous software polling - Hardware operates at line rate without software delays Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md index c4561a7f758..cd6c6af1588 100644 --- a/doc/qos/pfc_hardware_recovery_hld.md +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -95,6 +95,12 @@ Each phase of hardware detection and recovery happens based on the programming o - Configured values may differ from actual hardware-programmed values - Upper limits may apply based on platform (e.g., Broadcom: 1-15 × granularity) +**Hardware recovery advantages:** +- **Better accuracy in storm detection**: Hardware-based detection operates at line rate without software polling delays, providing more precise detection timing +- **Better accuracy in storm restoration**: Hardware-based restoration timing is more accurate as it doesn't depend on software polling intervals or system load +- **Reduced latency**: Hardware detection and recovery actions happen immediately without waiting for software polling cycles +- **Lower CPU overhead**: Eliminates continuous software polling, reducing CPU utilization and system load + #### 4.1.1 Hardware vs Software Recovery Flow The following diagram illustrates the decision flow for selecting hardware vs software recovery: From 5664c298e14b9e41feb0aaa74bbce368ffa14cb5 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:30:53 +0000 Subject: [PATCH 05/15] Remove non-existent Open Points section from TOC - Section 13 Open Points does not exist in the document - Cleaned up Table of Contents to match actual document structure Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md index cd6c6af1588..8072e76bd57 100644 --- a/doc/qos/pfc_hardware_recovery_hld.md +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -23,7 +23,6 @@ - [11. Restrictions/Limitations](#11-restrictionslimitations) - [12. Testing Requirements/Design](#12-testing-requirementsdesign) - [12.1 Additional Test Cases for Hardware Recovery](#121-additional-test-cases-for-hardware-recovery) -- [13. Open Points](#13-open-points) ## Revision From cac668825e53d36c256e1467fd1125b6c043e478 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:33:40 +0000 Subject: [PATCH 06/15] Add missing section 5.1 to Table of Contents - Added CLI Data Flow subsection to TOC for proper navigation - Fixes parsing/navigation issue in section 5 Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md index 8072e76bd57..4ffd5489a3a 100644 --- a/doc/qos/pfc_hardware_recovery_hld.md +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -12,6 +12,7 @@ - [4.1 Hardware Recovery Mechanism](#41-hardware-recovery-mechanism) - [4.2 PFC Watchdog Orchagent Refactoring](#42-pfc-watchdog-orchagent-refactoring) - [5. CLI Changes](#5-cli-changes) + - [5.1 CLI Data Flow](#51-cli-data-flow) - [6. SAI API](#6-sai-api) - [6.1 SAI Attributes](#61-sai-attributes) - [6.2 SAI Events](#62-sai-events) From acd91cc67e0e3ffa3320afc25560251bee334069 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 04:34:06 +0000 Subject: [PATCH 07/15] Fix mermaid diagram parsing in section 5.1 - Simplified the display table description in mermaid flowchart - Removed pipe characters that could cause parsing issues - Improved readability of the CLI data flow diagram Signed-off-by: Pinky Agrawal --- doc/qos/pfc_hardware_recovery_hld.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/qos/pfc_hardware_recovery_hld.md index 4ffd5489a3a..b36d895ffbd 100644 --- a/doc/qos/pfc_hardware_recovery_hld.md +++ b/doc/qos/pfc_hardware_recovery_hld.md @@ -255,7 +255,7 @@ flowchart LR G --> H D --> H - H --> I[Display Table:
PORT | RECOVERY TYPE | HW DETECTION TIME |
DETECTION GRANULARITY | HW RESTORATION TIME |
RESTORATION GRANULARITY] + H --> I[Display Table with:
PORT, RECOVERY TYPE, HW DETECTION TIME,
DETECTION GRANULARITY, HW RESTORATION TIME,
RESTORATION GRANULARITY] ``` ```shell From ee7c047394bf1594b21e7f1bfcd50655a2146f5d Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 16 Dec 2025 08:43:40 +0000 Subject: [PATCH 08/15] Organize PFC Watchdog HLD into dedicated directory - Created doc/pfcwd/ directory for PFC Watchdog documentation - Moved pfc_hardware_recovery_hld.md from doc/qos/ to doc/pfcwd/ - Better organization following SONiC documentation structure - Dedicated directory allows for future PFC Watchdog related documents Signed-off-by: Pinky Agrawal --- doc/{qos => pfcwd}/pfc_hardware_recovery_hld.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename doc/{qos => pfcwd}/pfc_hardware_recovery_hld.md (100%) diff --git a/doc/qos/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_hardware_recovery_hld.md similarity index 100% rename from doc/qos/pfc_hardware_recovery_hld.md rename to doc/pfcwd/pfc_hardware_recovery_hld.md From dcebc31a2cc059bd4925f6303d5f0c342ea2abe9 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Thu, 18 Dec 2025 08:38:41 +0000 Subject: [PATCH 09/15] Add STATUS column to CLI and update hardware recovery workflow - Added STATUS column to show pfcwd status command output to indicate success/failed programming status - Updated CLI data flow diagram to include status validation - Added example showing failed configuration due to unsupported timer ranges - Updated hardware recovery workflow diagram to show event notification happening immediately after storm detection - Added conditional logic for app_managed_recovery flag: - If true: wait for SAI_QUEUE_ATTR_PFC_DLR_INIT programming - If false: hardware automatically applies recovery action Signed-off-by: Pinky Agrawal --- doc/pfcwd/pfc_hardware_recovery_hld.md | 33 +++++++++++++++++--------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/doc/pfcwd/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_hardware_recovery_hld.md index b36d895ffbd..bad640002a9 100644 --- a/doc/pfcwd/pfc_hardware_recovery_hld.md +++ b/doc/pfcwd/pfc_hardware_recovery_hld.md @@ -1,7 +1,6 @@ # PFC Watchdog Hardware Recovery ## Table of Contents - - [Revision](#revision) - [Scope](#scope) - [Abbreviations](#abbreviations) @@ -29,7 +28,7 @@ | Rev | Date | Author | Change Description | |-----|------------|---------------|--------------------| -| 0.1 | 12/13/2024 | PFC Team | Initial version | +| 0.1 | 12/13/2025 | Pinky Agrawal | Initial version | ## Scope @@ -152,11 +151,20 @@ sequenceDiagram Note over HW: PFC Storm Occurs HW->>HW: Detect Storm (Hardware Timer) - HW->>HW: Apply Recovery Action HW->>SAI: Generate Event Notification SAI->>Orch: Storm Detected Event Orch->>DB: Update Storm Counters + alt app_managed_recovery = true + Note over Orch: Event handler sets
app_managed_recovery=true + Orch->>SAI: Program SAI_QUEUE_ATTR_PFC_DLR_INIT + SAI->>HW: Trigger Recovery Action + HW->>HW: Apply Recovery Action + else app_managed_recovery = false + Note over HW: Hardware automatically
applies recovery action + HW->>HW: Apply Recovery Action + end + Note over HW: Storm Subsides HW->>HW: Restoration Timer Expires HW->>HW: Restore Normal Operation @@ -230,10 +238,10 @@ There may be a granularity associated with the timers, so it is possible that th For example: if the hardware granularity is 100 ms and configured detection time through cli is 250 ms, then hardware might program either 200/300 ms. **Upper limit on the timer value** -Some platforms like broadcom only allow the timer value to be within the range <1-15> multiplied with the current granularity. So, it is possible that the value in the cli config cannot be programmed in the hardware. There is no mechanism to notify the config status today. +Some platforms like broadcom only allow the timer value to be within the range <1-15> multiplied with the current granularity. So, it is possible that the value in the cli config cannot be programmed in the hardware. **New CLI command** -We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. +We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, programming status, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. The STATUS column indicates whether the pfcwd configuration was successfully programmed (success/failed). Configuration can fail due to unsupported timer value ranges or hardware constraints. #### 5.1 CLI Data Flow @@ -249,22 +257,25 @@ flowchart LR C --> E[Get Actual HW Timer Values] C --> F[Get Platform Granularity] C --> G[Get Recovery Type] + C --> J[Validate Timer Ranges] E --> H[Format Display Data] F --> H G --> H + J --> H D --> H - H --> I[Display Table with:
PORT, RECOVERY TYPE, HW DETECTION TIME,
DETECTION GRANULARITY, HW RESTORATION TIME,
RESTORATION GRANULARITY] + H --> I[Display Table with:
PORT, RECOVERY TYPE, STATUS,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] ``` ```shell admin@sonic:~$ show pfcwd status -PORT RECOVERY TYPE HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ----------- ------------- ------------------- --------------------- --------------------- ----------------------- -Ethernet0 hardware 300 100ms 500 100ms -Ethernet4 software N/A N/A N/A N/A -Ethernet8 hardware 200 50ms 400 100ms +PORT RECOVERY TYPE STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY +---------- ------------- -------- ------------------- --------------------- --------------------- ----------------------- +Ethernet0 hardware success 300 100ms 500 100ms +Ethernet4 software success N/A N/A N/A N/A +Ethernet8 hardware success 200 50ms 400 100ms +Ethernet12 hardware failed N/A 100ms N/A 100ms ``` ## 6. SAI API From 013b692b9bf29cb7fd509bb528c1b07f7b0cc089 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Sun, 25 Jan 2026 09:51:12 +0000 Subject: [PATCH 10/15] Addressing review comments. Signed-off-by: Pinky Agrawal --- doc/pfcwd/pfc_hardware_recovery_hld.md | 52 ++++++++++++++++++++------ 1 file changed, 40 insertions(+), 12 deletions(-) diff --git a/doc/pfcwd/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_hardware_recovery_hld.md index bad640002a9..3ad53a99bcf 100644 --- a/doc/pfcwd/pfc_hardware_recovery_hld.md +++ b/doc/pfcwd/pfc_hardware_recovery_hld.md @@ -11,7 +11,8 @@ - [4.1 Hardware Recovery Mechanism](#41-hardware-recovery-mechanism) - [4.2 PFC Watchdog Orchagent Refactoring](#42-pfc-watchdog-orchagent-refactoring) - [5. CLI Changes](#5-cli-changes) - - [5.1 CLI Data Flow](#51-cli-data-flow) + - [5.1 New CLI command](#51-new-cli-command) + - [5.2 CLI Data Flow](#52-cli-data-flow) - [6. SAI API](#6-sai-api) - [6.1 SAI Attributes](#61-sai-attributes) - [6.2 SAI Events](#62-sai-events) @@ -240,10 +241,19 @@ For example: if the hardware granularity is 100 ms and configured detection time **Upper limit on the timer value** Some platforms like broadcom only allow the timer value to be within the range <1-15> multiplied with the current granularity. So, it is possible that the value in the cli config cannot be programmed in the hardware. -**New CLI command** +#### 5.1 New CLI command We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, programming status, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. The STATUS column indicates whether the pfcwd configuration was successfully programmed (success/failed). Configuration can fail due to unsupported timer value ranges or hardware constraints. -#### 5.1 CLI Data Flow +**Timer determination for hardware-based model** +For hardware-based recovery, the actual timer values programmed in hardware are determined through the following process: +1. **User Configuration**: User configures desired timer values through CLI (e.g., detection_time=250ms) +2. **Platform Capability Query**: Software queries SAI capability attributes to determine hardware granularity constraints (see section 6.1 for SAI capability attributes) +3. **Value Adjustment**: Software adjusts the configured value to the nearest supported hardware value based on granularity +4. **Hardware Programming**: Adjusted value is programmed via SAI attributes (`SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL`) +5. **Verification**: Software reads back the actual programmed value from hardware to display in `show pfcwd status` + + +#### 5.2 CLI Data Flow The following diagram illustrates how the new `show pfcwd status` command retrieves and displays information: @@ -268,14 +278,26 @@ flowchart LR H --> I[Display Table with:
PORT, RECOVERY TYPE, STATUS,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] ``` +**Example: ASIC with Hardware Recovery** + ```shell admin@sonic:~$ show pfcwd status PORT RECOVERY TYPE STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ---------- ------------- -------- ------------------- --------------------- --------------------- ----------------------- Ethernet0 hardware success 300 100ms 500 100ms -Ethernet4 software success N/A N/A N/A N/A Ethernet8 hardware success 200 50ms 400 100ms -Ethernet12 hardware failed N/A 100ms N/A 100ms +Ethernet12 hardware success 400 100ms 800 100ms +``` + +**Example: ASIC with Software Recovery** + +```shell +admin@sonic:~$ show pfcwd status +PORT RECOVERY TYPE STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY +---------- ------------- -------- ------------------- --------------------- --------------------- ----------------------- +Ethernet0 software success N/A N/A N/A N/A +Ethernet4 software success N/A N/A N/A N/A +Ethernet8 software success N/A N/A N/A N/A ``` ## 6. SAI API @@ -284,12 +306,19 @@ Following SAI statistics and attributes are used in this feature: ### 6.1 SAI Attributes -- `SAI_QUEUE_ATTR_ENABLE_PFC_DLDR` - Enable PFC deadlock detection and recovery -- `SAI_SWITCH_ATTR_QUEUE_PFC_DEADLOCK_NOTIFY` - Register callback for PFC deadlock events -- `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` - Detection timer intervals per port/per TC -- `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` - Recovery timer intervals per port/per TC -- `SAI_SWITCH_ATTR_PFC_DLR_PACKET_ACTION` - Configure drop/forward action -- `SAI_QUEUE_ATTR_PFC_DLR_INIT` - Manual recovery control +**Configuration Attributes:** + +| SAI Attribute | Description | +| ------------- | ----------- | +| `SAI_QUEUE_ATTR_ENABLE_PFC_DLDR` | Enable PFC deadlock detection and recovery | +| `SAI_SWITCH_ATTR_QUEUE_PFC_DEADLOCK_NOTIFY` | Register callback for PFC deadlock events | +| `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL_RANGE` | Query supported detection timer range per port | +| `SAI_PORT_ATTR_PFC_TC_DLD_TIMER_INTERVAL` | Set the granularity for the detection timer per port | +| `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` | Detection timer intervals per port/per TC | +| `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL_RANGE` | Query supported recovery timer range per port | +| `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` | Recovery timer intervals per port/per TC | +| `SAI_SWITCH_ATTR_PFC_DLR_PACKET_ACTION` | Configure drop/forward action | +| `SAI_QUEUE_ATTR_PFC_DLR_INIT` | Manual recovery control | ### 6.2 SAI Events @@ -350,7 +379,6 @@ All existing PFC watchdog testing remains unchanged as documented in the [PFC Wa - Validate hardware timer values and granularity information display - Test N/A display for software recovery ports - Verify column alignment and formatting -- Test command behavior on mixed hardware/software port configurations #### Hardware Recovery Functionality Testing - Verify hardware capability detection and automatic selection between hardware/software implementations From ecc08eacb6759a7fe84379df7eb11d9558beb896 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Mon, 2 Feb 2026 13:39:27 +0000 Subject: [PATCH 11/15] Add STATE_DB schema and SDK/hardware responsibilities to PFC Watchdog HLD This commit addresses review comments on PR #2159 by adding: 1. STATE_DB Schema (Section 7.1): - Added PFC_WD_STATE table to track detection/restoration events - Supports both hardware and software-based PFCWD methods - Includes programming_status field to indicate hardware programming success/failure - Stores operational state, timestamps, timer values, and granularity 2. Updated Diagrams: - Main flowchart now shows STATE_DB updates at initialization, detection, and restoration - Sequence diagram includes STATE_DB participant and state updates - CLI flow diagram queries STATE_DB for hardware-specific information 3. SDK and Hardware Responsibilities (Section 4.1.3): - Clarifies SDK responsibilities: register programming, capability reporting, event translation - Clarifies hardware responsibilities: line-rate detection, recovery actions, timer management Signed-off-by: Pinky Agrawal --- doc/pfcwd/pfc_hardware_recovery_hld.md | 178 +++++++++++++++++++++---- 1 file changed, 150 insertions(+), 28 deletions(-) diff --git a/doc/pfcwd/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_hardware_recovery_hld.md index 3ad53a99bcf..00794f47a63 100644 --- a/doc/pfcwd/pfc_hardware_recovery_hld.md +++ b/doc/pfcwd/pfc_hardware_recovery_hld.md @@ -9,6 +9,9 @@ - [3. Architecture Design](#3-architecture-design) - [4. High-Level Design](#4-high-level-design) - [4.1 Hardware Recovery Mechanism](#41-hardware-recovery-mechanism) + - [4.1.1 Hardware vs Software Recovery Flow](#411-hardware-vs-software-recovery-flow) + - [4.1.2 Hardware Recovery Workflow](#412-hardware-recovery-workflow) + - [4.1.3 SDK and Hardware Responsibilities](#413-sdk-and-hardware-responsibilities) - [4.2 PFC Watchdog Orchagent Refactoring](#42-pfc-watchdog-orchagent-refactoring) - [5. CLI Changes](#5-cli-changes) - [5.1 New CLI command](#51-new-cli-command) @@ -112,22 +115,32 @@ flowchart TD B -->|No| D[Initialize PfcWdOrch
Software Recovery] C --> E[Configure SAI Hardware
Watchdog Attributes] - E --> F[Hardware Detection
& Recovery Active] + E --> E1[Update STATE_DB:
status=operational
recovery_type=hardware] + E1 --> F[Hardware Detection
& Recovery Active] D --> G[Configure Lua Script
Polling & Handlers] - G --> H[Software Detection
& Recovery Active] + G --> G1[Update STATE_DB:
status=operational
recovery_type=software] + G1 --> H[Software Detection
& Recovery Active] F --> I[PFC Storm Detected
by Hardware] H --> J[PFC Storm Detected
by Lua Script] - I --> K[Hardware Automatic
Recovery Action] - J --> L[Software Handler
Recovery Action] + I --> I1[Update STATE_DB:
status=storm_detected
last_detection_time
detection_count++] + J --> J1[Update STATE_DB:
status=storm_detected
last_detection_time
detection_count++] + + I1 --> K[Hardware Automatic
Recovery Action] + J1 --> L[Software Handler
Recovery Action] K --> M[Hardware Event
Notification to SW] L --> N[Update Counters
& State] + M --> M1[Update STATE_DB:
status=storm_restored
last_restoration_time
storm_duration_ms] + N --> N1[Update STATE_DB:
status=storm_restored
last_restoration_time
storm_duration_ms] + M --> P[Software Counter
Polling & State Update] N --> O[Update Statistics
& CLI Display] + M1 --> O + N1 --> O P --> O ``` @@ -142,6 +155,7 @@ sequenceDiagram participant SAI as SAI Layer participant HW as Hardware participant DB as Counters DB + participant StateDB as STATE_DB CLI->>Orch: Configure PFC Watchdog Orch->>SAI: Set Detection Timer @@ -149,12 +163,14 @@ sequenceDiagram Orch->>SAI: Set Recovery Action Orch->>SAI: Enable Hardware Watchdog SAI->>HW: Program Hardware Timers + Orch->>StateDB: Update status=operational,
recovery_type=hardware,
detection_time_programmed,
granularity values Note over HW: PFC Storm Occurs HW->>HW: Detect Storm (Hardware Timer) HW->>SAI: Generate Event Notification SAI->>Orch: Storm Detected Event Orch->>DB: Update Storm Counters + Orch->>StateDB: Update status=storm_detected,
last_detection_time,
detection_count++ alt app_managed_recovery = true Note over Orch: Event handler sets
app_managed_recovery=true @@ -172,8 +188,31 @@ sequenceDiagram HW->>SAI: Generate Restoration Event SAI->>Orch: Storm Restored Event Orch->>DB: Update Restoration Counters + Orch->>StateDB: Update status=storm_restored,
last_restoration_time,
storm_duration_ms ``` +#### 4.1.3 SDK and Hardware Responsibilities + +In hardware-based PFC watchdog recovery, the responsibilities are distributed between the SDK layer and the hardware (ASIC). + +**SDK Responsibilities:** + +The SDK layer (e.g., Broadcom SDK) acts as the intermediary between SAI and the hardware. It is responsible for: +- Translating SAI attribute calls into hardware-specific register programming +- Reporting hardware capabilities such as supported timer ranges and granularity constraints +- Monitoring hardware event registers and translating them into SAI event callbacks (`SAI_QUEUE_PFC_DEADLOCK_EVENT_TYPE_DETECTED` and `SAI_QUEUE_PFC_DEADLOCK_EVENT_TYPE_RECOVERED`) +- Managing hardware resource allocation for timers, counters, and event queues +- Reading hardware counter registers and providing statistics via SAI APIs + +**Hardware (ASIC) Responsibilities:** + +The hardware performs the actual line-rate detection and recovery operations: +- Monitoring incoming PFC pause frames on each queue and tracking continuous pause state duration +- Comparing pause duration against programmed detection timers and generating detection events when timers expire +- Automatically applying recovery actions (drop/forward) to packets in affected queues +- Maintaining hardware timers for both detection and restoration at the configured granularity (e.g., 100ms increments) +- Generating hardware interrupts or events on storm detection and restoration + ### 4.2 PFC Watchdog Orchagent Refactoring This document outlines a proposal to refactor the PFC watchdog orchestrator architecture to support both hardware-based and software-based PFC recovery mechanisms. @@ -207,13 +246,31 @@ Orch (base class) `PfcWdHwOrch` is the new class that will handle hardware recovery mechanisms. There will be no handlers defined in hardware-based recovery unlike software recovery which defines drop and forward handlers. To update the counters periodically, `PfcWdHwOrch` itself will provide a function to read the required SAI attributes and get updated counters. +#### SKU-Based Override Mechanism + +In some cases, certain SKUs may need to use software-based PFCWD even when hardware capability is available. This can be due to: +- Hardware recovery in development mode +- Hardware limitations or bugs specific to certain SKU models +- Platform-specific requirements +- Testing or validation purposes + +For such cases, the orchagent can maintain a list of such SKUs and skip creating `PfcWdHwOrch` for those SKUs as given in the code below. + Runtime selection in `orchdaemon.cpp`: ```c // Check hardware capability bool pfcHwRecoverySupported = gSwitchOrch->checkPfcHwRecoverySupport(); -if (pfcHwRecoverySupported) +// Check if SKU is in the override list to force software-based PFCWD +std::vector swPfcwdSkuList = getSwPfcwdSkuOverrideList(); +std::string currentSku = gSwitchOrch->querySwitchSku(); +bool skuForcesSwPfcwd = std::find(swPfcwdSkuList.begin(), swPfcwdSkuList.end(), currentSku) != swPfcwdSkuList.end(); + +// Check hardware capability +bool pfcHwRecoverySupported = gSwitchOrch->checkPfcHwRecoverySupport(); + +if (pfcHwRecoverySupported && !skuForcesSwPfcwd) { SWSS_LOG_NOTICE("Starting hardware-based PFC watchdog (no handlers)"); m_orchList.push_back(new PfcWdHwOrch( @@ -259,45 +316,47 @@ The following diagram illustrates how the new `show pfcwd status` command retrie ```mermaid flowchart LR - A[show pfcwd status] --> B{Recovery Type?} + A[show pfcwd status] --> B[Query STATE_DB
PFC_WD_STATE Table] - B -->|Hardware| C[Query SAI Attributes] - B -->|Software| D[Return N/A Values] + B --> C[Read Per-Port/Queue Entries] - C --> E[Get Actual HW Timer Values] - C --> F[Get Platform Granularity] - C --> G[Get Recovery Type] - C --> J[Validate Timer Ranges] + C --> D[Extract recovery_type] + C --> E[Extract programming_status] + C --> F[Extract detection_time_programmed] + C --> G[Extract detection_time_granularity] + C --> H[Extract restoration_time_programmed] + C --> I[Extract restoration_time_granularity] - E --> H[Format Display Data] - F --> H - G --> H - J --> H - D --> H + D --> L[Format Display Data] + E --> L + F --> L + G --> L + H --> L + I --> L - H --> I[Display Table with:
PORT, RECOVERY TYPE, STATUS,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] + L --> M[Display Table with:
PORT, RECOVERY TYPE, PROGRAMMING_STATUS,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] ``` **Example: ASIC with Hardware Recovery** ```shell admin@sonic:~$ show pfcwd status -PORT RECOVERY TYPE STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ----------- ------------- -------- ------------------- --------------------- --------------------- ----------------------- -Ethernet0 hardware success 300 100ms 500 100ms -Ethernet8 hardware success 200 50ms 400 100ms -Ethernet12 hardware success 400 100ms 800 100ms +PORT RECOVERY TYPE PROGRAMMING_STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY +---------- ------------- ------------------ ------------------- --------------------- --------------------- ------------------------- +Ethernet0 hardware success 300 100ms 500 100ms +Ethernet4 hardware failed 2000 N/A N/A N/A +Ethernet12 hardware success 400 100ms 800 100ms ``` **Example: ASIC with Software Recovery** ```shell admin@sonic:~$ show pfcwd status -PORT RECOVERY TYPE STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ----------- ------------- -------- ------------------- --------------------- --------------------- ----------------------- -Ethernet0 software success N/A N/A N/A N/A -Ethernet4 software success N/A N/A N/A N/A -Ethernet8 software success N/A N/A N/A N/A +PORT RECOVERY TYPE PROGRAMMING_STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY +---------- ------------- ------------------ ------------------- --------------------- --------------------- ------------------------- +Ethernet0 software N/A N/A N/A N/A N/A +Ethernet4 software N/A N/A N/A N/A N/A +Ethernet8 software N/A N/A N/A N/A N/A ``` ## 6. SAI API @@ -342,6 +401,69 @@ The hardware-based PFC watchdog uses the same configuration interface as the exi - Runtime selection between hardware and software is automatic based on platform capabilities - No additional configuration parameters are required for hardware mode +### 7.1 STATE_DB Schema + +The following STATE_DB table is used to track PFCWD detection and restoration events for both hardware and software-based methods. This table serves as the data source for the `show pfcwd status` command. + +**PFC_WD_STATE Table:** + +``` +PFC_WD_STATE|| + "recovery_type": "hardware" | "software" + "programming_status": "success" | "failed" | "N/A" + "status": "operational" | "storm_detected" | "storm_restored" + "detection_count": + "last_detection_time": + "last_restoration_time": + "storm_duration_ms": + "detection_time_configured": + "detection_time_programmed": // N/A for software mode or if programming failed + "detection_time_granularity": // N/A for software mode or if programming failed + "restoration_time_configured": + "restoration_time_programmed": // N/A for software mode or if programming failed + "restoration_time_granularity": // N/A for software mode or if programming failed + "action": "drop" | "forward" +``` + +**Example Entries:** + +``` +PFC_WD_STATE|Ethernet0|3 + "recovery_type": "hardware" + "programming_status": "success" + "status": "storm_detected" + "detection_count": "5" + "last_detection_time": "2026-02-02T10:15:30Z" + "last_restoration_time": "2026-02-02T10:14:25Z" + "storm_duration_ms": "0" + "detection_time_configured": "250" + "detection_time_programmed": "300" + "detection_time_granularity": "100" + "restoration_time_configured": "450" + "restoration_time_programmed": "500" + "restoration_time_granularity": "100" + "action": "drop" +``` + +Software mode: +``` +PFC_WD_STATE|Ethernet8|3 + "recovery_type": "software" + "programming_status": "N/A" + "status": "operational" + "detection_count": "2" + "last_detection_time": "2026-02-02T09:30:15Z" + "last_restoration_time": "2026-02-02T09:30:20Z" + "storm_duration_ms": "5000" + "detection_time_configured": "400" + "detection_time_programmed": "N/A" + "detection_time_granularity": "N/A" + "restoration_time_configured": "400" + "restoration_time_programmed": "N/A" + "restoration_time_granularity": "N/A" + "action": "drop" +``` + ## 8. Manifest Not applicable - this is a core SONiC feature enhancement. From f203dcea6faf82220d21c83dca8bd9f05d01bbff Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Tue, 3 Feb 2026 10:41:52 +0000 Subject: [PATCH 12/15] Add output of show pfcwd config for clarity Signed-off-by: Pinky Agrawal --- doc/pfcwd/pfc_hardware_recovery_hld.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/doc/pfcwd/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_hardware_recovery_hld.md index 00794f47a63..c0ca32957ab 100644 --- a/doc/pfcwd/pfc_hardware_recovery_hld.md +++ b/doc/pfcwd/pfc_hardware_recovery_hld.md @@ -339,6 +339,19 @@ flowchart LR **Example: ASIC with Hardware Recovery** +First, let's view the configured values using the existing `show pfcwd config` command: + +```shell +admin@sonic:~$ show pfcwd config +PORT ACTION DETECTION TIME RESTORATION TIME +---------- -------- ---------------- ------------------ +Ethernet0 drop 350 550 +Ethernet4 drop 2000 2000 +Ethernet12 drop 400 800 +``` + +Now, let's use the new `show pfcwd status` command to see the actual hardware-programmed values and programming status: + ```shell admin@sonic:~$ show pfcwd status PORT RECOVERY TYPE PROGRAMMING_STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY @@ -348,6 +361,11 @@ Ethernet4 hardware failed 2000 N/A Ethernet12 hardware success 400 100ms 800 100ms ``` +**Key Observations:** +- **Ethernet0**: Configuration (350ms/550ms) was adjusted to (300ms/500ms) by hardware due to 100ms granularity. The `show pfcwd config` shows what the user requested (350/550), but `show pfcwd status` reveals the actual hardware-programmed values (300/500). +- **Ethernet4**: Configuration (2000ms/2000ms) **failed** to program because 2000ms exceeds hardware timer range. The `show pfcwd config` shows what the user requested, but `show pfcwd status` reveals the programming failure. +- **Ethernet12**: Configuration (400ms/800ms) successfully programmed to hardware. Granularity is 100ms. + **Example: ASIC with Software Recovery** ```shell From 6a99d4b41306e48253e814542454067f09ac43e5 Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Thu, 5 Feb 2026 11:31:01 +0000 Subject: [PATCH 13/15] Update based on review comments Signed-off-by: Pinky Agrawal --- doc/pfcwd/pfc_hardware_recovery_hld.md | 226 +++++++++++++++++++------ 1 file changed, 170 insertions(+), 56 deletions(-) diff --git a/doc/pfcwd/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_hardware_recovery_hld.md index c0ca32957ab..299eb487cec 100644 --- a/doc/pfcwd/pfc_hardware_recovery_hld.md +++ b/doc/pfcwd/pfc_hardware_recovery_hld.md @@ -33,6 +33,7 @@ | Rev | Date | Author | Change Description | |-----|------------|---------------|--------------------| | 0.1 | 12/13/2025 | Pinky Agrawal | Initial version | +| 0.2 | 02/05/2026 | Pinky Agrawal | Addressed review comments: Added configuration validation using SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE, updated SAI attributes section with extension attributes, clarified counter polling mechanism, updated show pfcwd status for software mode, added telemetry test cases | ## Scope @@ -244,7 +245,7 @@ Orch (base class) `PfcWdBaseOrch` is the new base class that will have the common logic of parsing the CLI and creating the tasks. This implementation exists in `PfcWdOrch` today and will be moved to this new class. `PfcWdOrch` will be removed and renamed to `PfcWdBaseOrch` without the template parameters. -`PfcWdHwOrch` is the new class that will handle hardware recovery mechanisms. There will be no handlers defined in hardware-based recovery unlike software recovery which defines drop and forward handlers. To update the counters periodically, `PfcWdHwOrch` itself will provide a function to read the required SAI attributes and get updated counters. +`PfcWdHwOrch` is the new class that will handle hardware recovery mechanisms. There will be no handlers defined in hardware-based recovery unlike software recovery which defines drop and forward handlers. To update the counters periodically, `PfcWdHwOrch` will use the same counter polling mechanism as software-based recovery. The counter querying remains the same as the software-based approach, using SAI queue statistics APIs (`SAI_QUEUE_STAT_PACKETS`, `SAI_QUEUE_STAT_DROPPED_PACKETS`, etc.) to read hardware counters and update COUNTERS_DB. #### SKU-Based Override Mechanism @@ -273,6 +274,46 @@ bool pfcHwRecoverySupported = gSwitchOrch->checkPfcHwRecoverySupport(); if (pfcHwRecoverySupported && !skuForcesSwPfcwd) { SWSS_LOG_NOTICE("Starting hardware-based PFC watchdog (no handlers)"); + + // Query hardware timer range capabilities during initialization + sai_attribute_t attr_dld, attr_dlr; + attr_dld.id = SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE; + attr_dlr.id = SAI_SWITCH_ATTR_PFC_TC_DLR_INTERVAL_RANGE; + + sai_status_t status_dld = sai_switch_api->get_switch_attribute(gSwitchId, 1, &attr_dld); + sai_status_t status_dlr = sai_switch_api->get_switch_attribute(gSwitchId, 1, &attr_dlr); + + if (status_dld == SAI_STATUS_SUCCESS && status_dlr == SAI_STATUS_SUCCESS) + { + uint32_t detection_min = attr_dld.value.u32range.min; + uint32_t detection_max = attr_dld.value.u32range.max; + uint32_t restoration_min = attr_dlr.value.u32range.min; + uint32_t restoration_max = attr_dlr.value.u32range.max; + + SWSS_LOG_NOTICE("PFC watchdog hardware detection timer range: %u - %u ms", + detection_min, detection_max); + SWSS_LOG_NOTICE("PFC watchdog hardware restoration timer range: %u - %u ms", + restoration_min, restoration_max); + + // Store hardware capabilities in STATE_DB for CLI validation + DBConnector stateDb("STATE_DB", 0); + Table capabilitiesTable(&stateDb, "PFC_WD_HW_CAPABILITIES"); + + vector fvVector; + fvVector.emplace_back("detection_timer_min", to_string(detection_min)); + fvVector.emplace_back("detection_timer_max", to_string(detection_max)); + fvVector.emplace_back("restoration_timer_min", to_string(restoration_min)); + fvVector.emplace_back("restoration_timer_max", to_string(restoration_max)); + fvVector.emplace_back("recovery_type", "hardware"); + + capabilitiesTable.set("GLOBAL", fvVector); + SWSS_LOG_NOTICE("Stored PFC watchdog hardware capabilities in STATE_DB"); + } + else + { + SWSS_LOG_WARN("Failed to query PFC watchdog hardware timer ranges"); + } + m_orchList.push_back(new PfcWdHwOrch( m_configDb, pfc_wd_tables, @@ -289,28 +330,66 @@ if (pfcHwRecoverySupported && !skuForcesSwPfcwd) All existing CLI commands for PFC watchdog will remain unchanged as documented in [section 2.7 of the PFC Watchdog Design](https://github.com/sonic-net/SONiC/wiki/PFC-Watchdog-Design#27-cli). -Hardware recovery might have some constraints as listed below: +### 5.1 Configuration Validation + +During orchagent initialization, the valid range for programming the detection and restoration intervals is queried using the SAI attribute `SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE`. This range is stored in STATE_DB and used to validate the `config pfcwd` command before programming the hardware. + +**Validation Process:** +1. **Initialization**: During orchagent startup, query `SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE` to get the hardware-supported timer range (min/max values) and store in STATE_DB +2. **Config Validation**: When user executes `config pfcwd start`, validate that the configured detection_time and restoration_time fall within the queried hardware range +3. **Rejection**: If the configured values are out of range, reject the configuration with an appropriate error message +4. **Programming**: Only valid configurations are programmed to hardware + +**STATE_DB Storage:** +The hardware timer range is stored in STATE_DB for access by CLI and other components: +``` +PFC_WD_HW_CAPABILITIES|GLOBAL + "detection_timer_min": + "detection_timer_max": + "restoration_timer_min": + "restoration_timer_max": + "recovery_type": "hardware" | "software" +``` +**Example:** +```bash +# Hardware supports range: 10ms - 1500ms +admin@sonic:~$ config pfcwd start --action drop --restoration-time 2000 Ethernet0 400 +Error: Restoration time 2000ms exceeds hardware maximum of 1500ms + +admin@sonic:~$ config pfcwd start --action drop --restoration-time 800 Ethernet0 400 +Success: PFC watchdog configured on Ethernet0 +``` + +**Hardware Constraints:** + +Even with validation, hardware may have additional constraints: **Timer granularity** -There may be a granularity associated with the timers, so it is possible that the value configured in h/w is different from the value configured through the cli. -For example: if the hardware granularity is 100 ms and configured detection time through cli is 250 ms, then hardware might program either 200/300 ms. +There may be a granularity associated with the timers, so it is possible that the value programmed in hardware is different from the value configured through the CLI. +For example: if the hardware granularity is 100 ms and configured detection time through CLI is 250 ms, then hardware might program either 200ms or 300ms. -**Upper limit on the timer value** -Some platforms like broadcom only allow the timer value to be within the range <1-15> multiplied with the current granularity. So, it is possible that the value in the cli config cannot be programmed in the hardware. +Since there is no SAI attribute to query the current granularity setting today, the `show pfcwd status` command is needed to display the actual hardware-programmed values and verify if they match the configured values. -#### 5.1 New CLI command -We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, programming status, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. The STATUS column indicates whether the pfcwd configuration was successfully programmed (success/failed). Configuration can fail due to unsupported timer value ranges or hardware constraints. +### 5.2 New CLI command +We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. + +**Purpose of this command:** +- Display the actual hardware-programmed timer values (which may differ from configured values due to granularity constraints) in h/w recovery mode. This command is N/A for s/w recovery mode. +- Since there is no SAI attribute to query the current granularity setting, users need visibility into what values were actually programmed in hardware **Timer determination for hardware-based model** For hardware-based recovery, the actual timer values programmed in hardware are determined through the following process: 1. **User Configuration**: User configures desired timer values through CLI (e.g., detection_time=250ms) -2. **Platform Capability Query**: Software queries SAI capability attributes to determine hardware granularity constraints (see section 6.1 for SAI capability attributes) -3. **Value Adjustment**: Software adjusts the configured value to the nearest supported hardware value based on granularity -4. **Hardware Programming**: Adjusted value is programmed via SAI attributes (`SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL`) -5. **Verification**: Software reads back the actual programmed value from hardware to display in `show pfcwd status` +2. **Range Validation**: Software validates the configured value against the queried hardware range (`SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE`) +3. **Granularity Selection**: Software determines the appropriate granularity (1ms, 10ms, or 100ms) based on the configured value +4. **Value Adjustment**: Software adjusts the configured value to the nearest multiple of the selected granularity (e.g., 250ms with 100ms granularity → 200ms or 300ms) +5. **Hardware Programming**: + - First, set the granularity using `SAI_SWITCH_ATTR_PFC_TC_DLD_TIMER_INTERVAL` or `SAI_PORT_ATTR_PFC_TC_DLD_TIMER_INTERVAL` + - Then, program the timer value as `configured_value / granularity` using `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` +6. **Verification**: Software reads back the actual programmed value from hardware to display in `show pfcwd status` -#### 5.2 CLI Data Flow +### 5.3 CLI Data Flow The following diagram illustrates how the new `show pfcwd status` command retrieves and displays information: @@ -321,20 +400,18 @@ flowchart LR B --> C[Read Per-Port/Queue Entries] C --> D[Extract recovery_type] - C --> E[Extract programming_status] - C --> F[Extract detection_time_programmed] - C --> G[Extract detection_time_granularity] - C --> H[Extract restoration_time_programmed] - C --> I[Extract restoration_time_granularity] + C --> E[Extract detection_time_programmed] + C --> F[Extract detection_time_granularity] + C --> G[Extract restoration_time_programmed] + C --> H[Extract restoration_time_granularity] D --> L[Format Display Data] E --> L F --> L G --> L H --> L - I --> L - L --> M[Display Table with:
PORT, RECOVERY TYPE, PROGRAMMING_STATUS,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] + L --> M[Display Table with:
PORT, RECOVERY TYPE,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] ``` **Example: ASIC with Hardware Recovery** @@ -346,35 +423,32 @@ admin@sonic:~$ show pfcwd config PORT ACTION DETECTION TIME RESTORATION TIME ---------- -------- ---------------- ------------------ Ethernet0 drop 350 550 -Ethernet4 drop 2000 2000 Ethernet12 drop 400 800 ``` -Now, let's use the new `show pfcwd status` command to see the actual hardware-programmed values and programming status: +Now, let's use the new `show pfcwd status` command to see the actual hardware-programmed values: ```shell admin@sonic:~$ show pfcwd status -PORT RECOVERY TYPE PROGRAMMING_STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ----------- ------------- ------------------ ------------------- --------------------- --------------------- ------------------------- -Ethernet0 hardware success 300 100ms 500 100ms -Ethernet4 hardware failed 2000 N/A N/A N/A -Ethernet12 hardware success 400 100ms 800 100ms +PORT RECOVERY TYPE HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY +---------- ------------- ------------------- --------------------- --------------------- ------------------------- +Ethernet0 hardware 300 100ms 500 100ms +Ethernet12 hardware 400 100ms 800 100ms ``` **Key Observations:** - **Ethernet0**: Configuration (350ms/550ms) was adjusted to (300ms/500ms) by hardware due to 100ms granularity. The `show pfcwd config` shows what the user requested (350/550), but `show pfcwd status` reveals the actual hardware-programmed values (300/500). -- **Ethernet4**: Configuration (2000ms/2000ms) **failed** to program because 2000ms exceeds hardware timer range. The `show pfcwd config` shows what the user requested, but `show pfcwd status` reveals the programming failure. - **Ethernet12**: Configuration (400ms/800ms) successfully programmed to hardware. Granularity is 100ms. +**Note**: Since configuration validation is performed at both CLI and orchagent layers (see section 5.1), invalid configurations (e.g., values exceeding hardware timer range) are rejected before programming, ensuring all entries in `show pfcwd status` represent successfully programmed configurations. + **Example: ASIC with Software Recovery** +For platforms using software-based PFC watchdog recovery, the `show pfcwd status` command is not applicable since there are no hardware-programmed values or granularity constraints to display. + ```shell admin@sonic:~$ show pfcwd status -PORT RECOVERY TYPE PROGRAMMING_STATUS HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ----------- ------------- ------------------ ------------------- --------------------- --------------------- ------------------------- -Ethernet0 software N/A N/A N/A N/A N/A -Ethernet4 software N/A N/A N/A N/A N/A -Ethernet8 software N/A N/A N/A N/A N/A +This command is not applicable for software-based PFC watchdog recovery mode. ``` ## 6. SAI API @@ -383,19 +457,37 @@ Following SAI statistics and attributes are used in this feature: ### 6.1 SAI Attributes -**Configuration Attributes:** +**Queue Attributes (SAI_OBJECT_TYPE_QUEUE):** + +| SAI Attribute | Type | Flags | Description | +| ------------- | ---- | ----- | ----------- | +| `SAI_QUEUE_ATTR_ENABLE_PFC_DLDR` | bool | CREATE_AND_SET | Enable PFC deadlock detection and recovery on a lossless queue | +| `SAI_QUEUE_ATTR_PFC_DLR_INIT` | bool | CREATE_AND_SET | Start/stop PFC deadlock recovery manually (app-managed recovery) | + +**Port Attributes (SAI_OBJECT_TYPE_PORT):** + +| SAI Attribute | Type | Flags | Description | +| ------------- | ---- | ----- | ----------- | +| `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL_RANGE` | sai_u32_range_t | READ_ONLY | Query supported detection timer range per port (capability query) | +| `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` | sai_map_list_t | CREATE_AND_SET | Detection timer intervals per port/per TC in milliseconds | +| `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL_RANGE` | sai_u32_range_t | READ_ONLY | Query supported recovery timer range per port (capability query) | +| `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` | sai_map_list_t | CREATE_AND_SET | Recovery timer intervals per port/per TC in milliseconds | +| `SAI_PORT_ATTR_PFC_TC_DLD_TIMER_INTERVAL` (Extension) | sai_map_list_t | CREATE_AND_SET | Port PFC Deadlock Detection timer granularity of all PFC priorities | -| SAI Attribute | Description | -| ------------- | ----------- | -| `SAI_QUEUE_ATTR_ENABLE_PFC_DLDR` | Enable PFC deadlock detection and recovery | -| `SAI_SWITCH_ATTR_QUEUE_PFC_DEADLOCK_NOTIFY` | Register callback for PFC deadlock events | -| `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL_RANGE` | Query supported detection timer range per port | -| `SAI_PORT_ATTR_PFC_TC_DLD_TIMER_INTERVAL` | Set the granularity for the detection timer per port | -| `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` | Detection timer intervals per port/per TC | -| `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL_RANGE` | Query supported recovery timer range per port | -| `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` | Recovery timer intervals per port/per TC | -| `SAI_SWITCH_ATTR_PFC_DLR_PACKET_ACTION` | Configure drop/forward action | -| `SAI_QUEUE_ATTR_PFC_DLR_INIT` | Manual recovery control | +**Switch Attributes (SAI_OBJECT_TYPE_SWITCH):** + +| SAI Attribute | Type | Flags | Description | +| ------------- | ---- | ----- | ----------- | +| `SAI_SWITCH_ATTR_QUEUE_PFC_DEADLOCK_NOTIFY` | sai_pointer_t | CREATE_AND_SET | Register callback for PFC deadlock events | +| `SAI_SWITCH_ATTR_PFC_DLR_PACKET_ACTION` | sai_packet_action_t | CREATE_AND_SET | Configure drop/forward action during recovery | +| `SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE` | sai_u32_range_t | READ_ONLY | Query supported detection timer range at switch level (capability query) | +| `SAI_SWITCH_ATTR_PFC_TC_DLD_TIMER_INTERVAL` (Extension) | sai_map_list_t | CREATE_AND_SET | PFC Deadlock Detection timer granularity in milliseconds (1ms, 10ms, or 100ms) | + +**Notes:** +- Extension attributes are defined in `saiportextensions.h` and `saiswitchextensions.h` +- The `*_RANGE` attributes are READ_ONLY and used to query hardware capabilities (min/max timer values) +- The `*_TIMER_INTERVAL` attributes are used to SET granularity values, not query them +- There is no READ_ONLY attribute to query the current granularity setting; applications must track what they configured ### 6.2 SAI Events @@ -421,39 +513,42 @@ The hardware-based PFC watchdog uses the same configuration interface as the exi ### 7.1 STATE_DB Schema -The following STATE_DB table is used to track PFCWD detection and restoration events for both hardware and software-based methods. This table serves as the data source for the `show pfcwd status` command. +The following STATE_DB table is used to track PFCWD detection and restoration events for both hardware and software-based methods. This table serves as the data source for the `show pfcwd status` command and telemetry reporting. **PFC_WD_STATE Table:** +This table tracks the current state and configuration of PFC watchdog on each port/queue. + ``` PFC_WD_STATE|| "recovery_type": "hardware" | "software" - "programming_status": "success" | "failed" | "N/A" "status": "operational" | "storm_detected" | "storm_restored" "detection_count": + "restoration_count": "last_detection_time": "last_restoration_time": "storm_duration_ms": "detection_time_configured": - "detection_time_programmed": // N/A for software mode or if programming failed - "detection_time_granularity": // N/A for software mode or if programming failed + "detection_time_programmed": // N/A for software mode + "detection_time_granularity": // N/A for software mode "restoration_time_configured": - "restoration_time_programmed": // N/A for software mode or if programming failed - "restoration_time_granularity": // N/A for software mode or if programming failed + "restoration_time_programmed": // N/A for software mode + "restoration_time_granularity": // N/A for software mode "action": "drop" | "forward" ``` **Example Entries:** +PFC_WD_STATE table (Hardware mode): ``` PFC_WD_STATE|Ethernet0|3 "recovery_type": "hardware" - "programming_status": "success" "status": "storm_detected" "detection_count": "5" + "restoration_count": "4" "last_detection_time": "2026-02-02T10:15:30Z" "last_restoration_time": "2026-02-02T10:14:25Z" - "storm_duration_ms": "0" + "storm_duration_ms": "5000" "detection_time_configured": "250" "detection_time_programmed": "300" "detection_time_granularity": "100" @@ -463,13 +558,13 @@ PFC_WD_STATE|Ethernet0|3 "action": "drop" ``` -Software mode: +PFC_WD_STATE table (Software mode): ``` PFC_WD_STATE|Ethernet8|3 "recovery_type": "software" - "programming_status": "N/A" "status": "operational" "detection_count": "2" + "restoration_count": "2" "last_detection_time": "2026-02-02T09:30:15Z" "last_restoration_time": "2026-02-02T09:30:20Z" "storm_duration_ms": "5000" @@ -520,8 +615,27 @@ All existing PFC watchdog testing remains unchanged as documented in the [PFC Wa - Test N/A display for software recovery ports - Verify column alignment and formatting +#### Configuration Validation Testing +- Verify that `config pfcwd start` validates timer values against hardware range queried via `SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE` +- Test rejection of out-of-range timer values with appropriate error messages +- Validate that only in-range configurations are accepted and programmed to hardware +- Test edge cases (minimum value, maximum value, just below minimum, just above maximum) + #### Hardware Recovery Functionality Testing - Verify hardware capability detection and automatic selection between hardware/software implementations - Test SAI attribute programming for hardware watchdog configuration - Validate timer granularity constraints and hardware programming -- Test error handling for unsupported timer values and hardware programming failures \ No newline at end of file +- Verify that granularity is set correctly before programming timer values + +#### Telemetry Testing +- **Test STATE_DB telemetry reporting via gNMI/streaming telemetry** + - Subscribe to `PFC_WD_STATE` table updates via gNMI + - Trigger a PFC storm and verify detection event is reported via telemetry + - Verify restoration event is reported via telemetry + - Validate all fields are correctly mapped in telemetry data model + - Test telemetry reporting for both hardware and software recovery modes + - Verify detection_count and restoration_count are correctly incremented +- **Test telemetry data model mapping** + - Verify STATE_DB fields map correctly to OpenConfig or SONiC YANG paths + - Test telemetry subscription filters (per-port, per-queue) + - Validate telemetry data types and encoding \ No newline at end of file From d72c84091683760dd4b8ccbd841d9015eddb3ebc Mon Sep 17 00:00:00 2001 From: Pinky Agrawal Date: Fri, 17 Apr 2026 14:26:07 +0530 Subject: [PATCH 14/15] Address eddyk-nvidia review comments on PFC watchdog HLD - Rename document to pfc_watchdog_hardware_detection_recovery.md to reflect both detection and recovery - Update document title to 'PFC Watchdog Hardware-Based Detection and Recovery' - Add comprehensive vendor-specific implementation note in Scope section - Remove Broadcom-specific section 4.1.3 (SDK and Hardware Responsibilities) - Remove extension SAI attributes (SAI_*_TIMER_INTERVAL) - Add missing switch-level SAI attributes (SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL, SAI_SWITCH_ATTR_PFC_TC_DLR_INTERVAL) - Update 'show pfcwd status' command output format to include mode, supported ranges, and programming status - Add PFC_WD_GLOBAL STATE_DB table for global capabilities - Update PFC_WD_STATE table with programming_status field - Remove specific timer calculation examples and make all references vendor-agnostic - Simplify flowchart node labels by removing HTML
tags for better rendering - Update CLI data flow diagram to reflect new output structure Addresses review comments from eddyk-nvidia regarding: - Terminology (detection & recovery vs just recovery) - Vendor-specific implementation flexibility - Timer calculations being vendor-specific - SAI attribute completeness - CONFIG_DB overr- CONFIG_DB overr- CONFIG_DB overr- CONFIG_DB overr- CONFIG_DB overr- Cnky Agrawal Signed-off-by: Pinky Agrawal --- ...c_watchdog_hardware_detection_recovery.md} | 273 ++++++++++-------- 1 file changed, 150 insertions(+), 123 deletions(-) rename doc/pfcwd/{pfc_hardware_recovery_hld.md => pfc_watchdog_hardware_detection_recovery.md} (70%) diff --git a/doc/pfcwd/pfc_hardware_recovery_hld.md b/doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md similarity index 70% rename from doc/pfcwd/pfc_hardware_recovery_hld.md rename to doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md index 299eb487cec..55e4a16dd68 100644 --- a/doc/pfcwd/pfc_hardware_recovery_hld.md +++ b/doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md @@ -1,43 +1,63 @@ -# PFC Watchdog Hardware Recovery +# PFC Watchdog Hardware-Based Detection and Recovery ## Table of Contents -- [Revision](#revision) -- [Scope](#scope) -- [Abbreviations](#abbreviations) -- [1. Overview](#1-overview) -- [2. Requirements](#2-requirements) -- [3. Architecture Design](#3-architecture-design) -- [4. High-Level Design](#4-high-level-design) - - [4.1 Hardware Recovery Mechanism](#41-hardware-recovery-mechanism) - - [4.1.1 Hardware vs Software Recovery Flow](#411-hardware-vs-software-recovery-flow) - - [4.1.2 Hardware Recovery Workflow](#412-hardware-recovery-workflow) - - [4.1.3 SDK and Hardware Responsibilities](#413-sdk-and-hardware-responsibilities) - - [4.2 PFC Watchdog Orchagent Refactoring](#42-pfc-watchdog-orchagent-refactoring) -- [5. CLI Changes](#5-cli-changes) - - [5.1 New CLI command](#51-new-cli-command) - - [5.2 CLI Data Flow](#52-cli-data-flow) -- [6. SAI API](#6-sai-api) - - [6.1 SAI Attributes](#61-sai-attributes) - - [6.2 SAI Events](#62-sai-events) - - [6.3 SAI Statistics](#63-sai-statistics) -- [7. Configuration and Management](#7-configuration-and-management) -- [8. Manifest](#8-manifest) -- [9. CLI-YANG Model Enhancements](#9-cli-yang-model-enhancements) -- [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) -- [11. Restrictions/Limitations](#11-restrictionslimitations) -- [12. Testing Requirements/Design](#12-testing-requirementsdesign) - - [12.1 Additional Test Cases for Hardware Recovery](#121-additional-test-cases-for-hardware-recovery) +- [PFC Watchdog Hardware-Based Detection and Recovery](#pfc-watchdog-hardware-based-detection-and-recovery) + - [Table of Contents](#table-of-contents) + - [Revision](#revision) + - [Scope](#scope) + - [Abbreviations](#abbreviations) + - [1. Overview](#1-overview) + - [2. Requirements](#2-requirements) + - [3. Architecture Design](#3-architecture-design) + - [4. High-Level Design](#4-high-level-design) + - [4.1 Hardware Recovery Mechanism](#41-hardware-recovery-mechanism) + - [4.1.1 Hardware vs Software Recovery Flow](#411-hardware-vs-software-recovery-flow) + - [4.1.2 Hardware Recovery Workflow](#412-hardware-recovery-workflow) + - [4.2 PFC Watchdog Orchagent Refactoring](#42-pfc-watchdog-orchagent-refactoring) + - [Design Goals](#design-goals) + - [Current Architecture](#current-architecture) + - [Proposed Architecture](#proposed-architecture) + - [Implementation Details](#implementation-details) + - [SKU-Based Override Mechanism](#sku-based-override-mechanism) + - [5. CLI Changes](#5-cli-changes) + - [5.1 Configuration Validation](#51-configuration-validation) + - [5.2 New CLI command](#52-new-cli-command) + - [5.3 CLI Data Flow](#53-cli-data-flow) + - [6. SAI API](#6-sai-api) + - [6.1 SAI Attributes](#61-sai-attributes) + - [6.2 SAI Events](#62-sai-events) + - [6.3 SAI Statistics](#63-sai-statistics) + - [7. Configuration and Management](#7-configuration-and-management) + - [7.1 STATE\_DB Schema](#71-state_db-schema) + - [8. Manifest](#8-manifest) + - [9. CLI-YANG Model Enhancements](#9-cli-yang-model-enhancements) + - [10. Warmboot and Fastboot Design Impact](#10-warmboot-and-fastboot-design-impact) + - [11. Restrictions/Limitations](#11-restrictionslimitations) + - [12. Testing Requirements/Design](#12-testing-requirementsdesign) + - [12.1 Additional Test Cases for Hardware Recovery](#121-additional-test-cases-for-hardware-recovery) + - [New CLI Command Testing](#new-cli-command-testing) + - [Configuration Validation Testing](#configuration-validation-testing) + - [Hardware Recovery Functionality Testing](#hardware-recovery-functionality-testing) + - [Telemetry Testing](#telemetry-testing) ## Revision | Rev | Date | Author | Change Description | |-----|------------|---------------|--------------------| | 0.1 | 12/13/2025 | Pinky Agrawal | Initial version | -| 0.2 | 02/05/2026 | Pinky Agrawal | Addressed review comments: Added configuration validation using SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE, updated SAI attributes section with extension attributes, clarified counter polling mechanism, updated show pfcwd status for software mode, added telemetry test cases | +| 0.2 | 02/05/2026 | Pinky Agrawal | Addressed review comments: Added configuration validation using SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE, clarified counter polling mechanism, updated show pfcwd status for software mode, added telemetry test cases | ## Scope -This document describes the design for hardware-based PFC (Priority Flow Control) watchdog recovery mechanism in SONiC. It covers the extension of existing software-based PFC watchdog functionality to support hardware-accelerated detection and recovery. +This document describes the design for hardware-based PFC (Priority Flow Control) watchdog detection and recovery mechanism in SONiC. It covers the extension of existing software-based PFC watchdog functionality to support hardware-accelerated detection and recovery. + +The term "hardware-based" (or "H/W") in this document refers to capabilities exposed through the SAI API layer. The actual implementation is vendor-specific and can be realized in various ways: +- Fully in hardware (ASIC) +- In ASIC firmware +- In the vendor SAI implementation +- Any combination of the above (e.g., detection in hardware/firmware and recovery in vendor SAI) + +Throughout this document, wherever "H/W", "hardware", "ASIC", or similar terms are used to describe detection, recovery, or other operations, they refer to this vendor-specific implementation layer. This document focuses on the SONiC perspective and interactions with the SAI layer, without prescribing specific vendor implementation details. ## Abbreviations @@ -66,8 +86,8 @@ The hardware-based PFC watchdog recovery feature shall provide: 2. **Hardware Recovery**: Automatic recovery actions (drop/forward) performed by hardware 3. **Backward Compatibility**: Existing software-based PFC watchdog functionality remains unchanged 4. **Runtime Selection**: Automatic selection between hardware and software implementations based on platform capabilities -5. **Enhanced CLI**: New `show pfcwd status` command to display recovery type, actual hardware timer values, and granularity information -6. **Error Handling**: Proper validation and error reporting for hardware timer granularity constraints +5. **Enhanced CLI**: New `show pfcwd status` command to display recovery type, programming status, actual hardware timer values +6. **Error Handling**: Proper validation and error reporting for programming based on SAI errors ## 3. Architecture Design @@ -89,15 +109,15 @@ Each phase of hardware detection and recovery happens based on the programming o | ----- | ----- | ----- | | Deadlock detection requirement | Polling period for lua script configured in S/W | Enable h/w deadlock detection and recovery. Configure detection timer. Register callback for h/w events | | Detection mechanism | Lua script polls the queue to check if queue received PFC frames continuously for detection time | H/w detects the deadlock and publishes an event to the s/w to take action, if required. | -| PFC pause frame handling | Software disables PFC by clearing the PFC mask bit for the affected queue (ignores incoming PFC pause frames) | Hardware sets the registers that ignores PFC pause frames for the affected queue during storm mitigation | +| PFC pause frame handling | Software disables PFC by clearing the PFC mask bit for the affected queue (ignores incoming PFC pause frames) | H/W ignores PFC pause frames for the affected queue during storm mitigation. No action needed in orchagent. | | Mitigation | Orchagent programs ACLS to take drop action or programs a zero-buffer profile | H/w takes the drop/forward action | | Recovery requirement | Polling period for lua script configured in S/W | Configure recovery timer attribute | -| Recovery | Lua script polls if the queue remains unpaused for restoration-time amount of time to recover | H/w recovers the queue and publishes an event to the s/w to take action, if required. | +| Recovery | Lua script polls if the queue remains unpaused for restoration-time amount of time to recover | H/W recovers the queue and publishes an event to the s/w when recovery completes. | **Timer granularity constraints:** -- Hardware may have granularity limitations (e.g., 100ms granularity) +- Hardware may have vendor-specific granularity limitations - Configured values may differ from actual hardware-programmed values -- Upper limits may apply based on platform (e.g., Broadcom: 1-15 × granularity) +- Timer value calculations and constraints are vendor-specific **Hardware recovery advantages:** - **Better accuracy in storm detection**: Hardware-based detection operates at line rate without software polling delays, providing more precise detection timing @@ -111,35 +131,35 @@ The following diagram illustrates the decision flow for selecting hardware vs so ```mermaid flowchart TD - A[PFC Watchdog Configuration] --> B{Platform Supports
Hardware Recovery?} + A[PFC Watchdog Configuration] --> B{Platform Supports Hardware Recovery?} B -->|Yes| C[Initialize PfcWdHwOrch] - B -->|No| D[Initialize PfcWdOrch
Software Recovery] + B -->|No| D[Initialize PfcWdOrch Software Recovery] - C --> E[Configure SAI Hardware
Watchdog Attributes] - E --> E1[Update STATE_DB:
status=operational
recovery_type=hardware] - E1 --> F[Hardware Detection
& Recovery Active] + C --> E[Configure SAI Hardware Watchdog Attributes] + E --> E1[Update STATE_DB: operational, hardware] + E1 --> F[Hardware Detection & Recovery Active] - D --> G[Configure Lua Script
Polling & Handlers] - G --> G1[Update STATE_DB:
status=operational
recovery_type=software] - G1 --> H[Software Detection
& Recovery Active] + D --> G[Configure Lua Script Polling & Handlers] + G --> G1[Update STATE_DB: operational, software] + G1 --> H[Software Detection & Recovery Active] - F --> I[PFC Storm Detected
by Hardware] - H --> J[PFC Storm Detected
by Lua Script] + F --> I[PFC Storm Detected by Hardware] + H --> J[PFC Storm Detected by Lua Script] - I --> I1[Update STATE_DB:
status=storm_detected
last_detection_time
detection_count++] - J --> J1[Update STATE_DB:
status=storm_detected
last_detection_time
detection_count++] + I --> I1[Update STATE_DB: storm_detected] + J --> J1[Update STATE_DB: storm_detected] - I1 --> K[Hardware Automatic
Recovery Action] - J1 --> L[Software Handler
Recovery Action] + I1 --> K[Hardware Automatic Recovery Action] + J1 --> L[Software Handler Recovery Action] - K --> M[Hardware Event
Notification to SW] - L --> N[Update Counters
& State] + K --> M[Hardware Event Notification to SW] + L --> N[Update Counters & State] - M --> M1[Update STATE_DB:
status=storm_restored
last_restoration_time
storm_duration_ms] - N --> N1[Update STATE_DB:
status=storm_restored
last_restoration_time
storm_duration_ms] + M --> M1[Update STATE_DB: storm_restored] + N --> N1[Update STATE_DB: storm_restored] - M --> P[Software Counter
Polling & State Update] - N --> O[Update Statistics
& CLI Display] + M --> P[Software Counter Polling & State Update] + N --> O[Update Statistics & CLI Display] M1 --> O N1 --> O P --> O @@ -192,28 +212,6 @@ sequenceDiagram Orch->>StateDB: Update status=storm_restored,
last_restoration_time,
storm_duration_ms ``` -#### 4.1.3 SDK and Hardware Responsibilities - -In hardware-based PFC watchdog recovery, the responsibilities are distributed between the SDK layer and the hardware (ASIC). - -**SDK Responsibilities:** - -The SDK layer (e.g., Broadcom SDK) acts as the intermediary between SAI and the hardware. It is responsible for: -- Translating SAI attribute calls into hardware-specific register programming -- Reporting hardware capabilities such as supported timer ranges and granularity constraints -- Monitoring hardware event registers and translating them into SAI event callbacks (`SAI_QUEUE_PFC_DEADLOCK_EVENT_TYPE_DETECTED` and `SAI_QUEUE_PFC_DEADLOCK_EVENT_TYPE_RECOVERED`) -- Managing hardware resource allocation for timers, counters, and event queues -- Reading hardware counter registers and providing statistics via SAI APIs - -**Hardware (ASIC) Responsibilities:** - -The hardware performs the actual line-rate detection and recovery operations: -- Monitoring incoming PFC pause frames on each queue and tracking continuous pause state duration -- Comparing pause duration against programmed detection timers and generating detection events when timers expire -- Automatically applying recovery actions (drop/forward) to packets in affected queues -- Maintaining hardware timers for both detection and restoration at the configured granularity (e.g., 100ms increments) -- Generating hardware interrupts or events on storm detection and restoration - ### 4.2 PFC Watchdog Orchagent Refactoring This document outlines a proposal to refactor the PFC watchdog orchestrator architecture to support both hardware-based and software-based PFC recovery mechanisms. @@ -365,28 +363,32 @@ Success: PFC watchdog configured on Ethernet0 Even with validation, hardware may have additional constraints: **Timer granularity** -There may be a granularity associated with the timers, so it is possible that the value programmed in hardware is different from the value configured through the CLI. -For example: if the hardware granularity is 100 ms and configured detection time through CLI is 250 ms, then hardware might program either 200ms or 300ms. +The actual value programmed in hardware may differ from the value configured through the CLI due to vendor-specific granularity constraints and timer implementation details. Since there is no SAI attribute to query the current granularity setting today, the `show pfcwd status` command is needed to display the actual hardware-programmed values and verify if they match the configured values. ### 5.2 New CLI command -We propose to add a new CLI command `show pfcwd status` to display hardware-specific information including recovery type, hardware detection time, hardware restoration time, detection time granularity, and restoration time granularity. +We propose to add a new CLI command `show pfcwd status` to display hardware-specific operational information including: +- PFC watchdog mode (hardware or software) +- Supported hardware timer ranges (detection and restoration intervals) +- Per-port programming status +- Actual hardware-programmed timer values **Purpose of this command:** -- Display the actual hardware-programmed timer values (which may differ from configured values due to granularity constraints) in h/w recovery mode. This command is N/A for s/w recovery mode. -- Since there is no SAI attribute to query the current granularity setting, users need visibility into what values were actually programmed in hardware +- Display the PFC watchdog mode in use (hardware or software) +- Show supported hardware timer ranges queried from SAI capabilities +- Display programming status per port (success or failed) +- Show the actual hardware-programmed timer values (which may differ from configured values due to vendor-specific constraints) **Timer determination for hardware-based model** For hardware-based recovery, the actual timer values programmed in hardware are determined through the following process: -1. **User Configuration**: User configures desired timer values through CLI (e.g., detection_time=250ms) +1. **User Configuration**: User configures desired timer values through CLI 2. **Range Validation**: Software validates the configured value against the queried hardware range (`SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE`) -3. **Granularity Selection**: Software determines the appropriate granularity (1ms, 10ms, or 100ms) based on the configured value -4. **Value Adjustment**: Software adjusts the configured value to the nearest multiple of the selected granularity (e.g., 250ms with 100ms granularity → 200ms or 300ms) -5. **Hardware Programming**: - - First, set the granularity using `SAI_SWITCH_ATTR_PFC_TC_DLD_TIMER_INTERVAL` or `SAI_PORT_ATTR_PFC_TC_DLD_TIMER_INTERVAL` - - Then, program the timer value as `configured_value / granularity` using `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` -6. **Verification**: Software reads back the actual programmed value from hardware to display in `show pfcwd status` +3. **Value Adjustment**: The actual programmed value may differ from the configured value based on vendor-specific hardware constraints and granularity requirements +4. **Hardware Programming**: Program the timer value using `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` +5. **Verification**: Software reads back the actual programmed value from hardware to display in `show pfcwd status` + +**Note:** The timer value calculation and adjustment is vendor-specific and depends on the hardware implementation. ### 5.3 CLI Data Flow @@ -394,24 +396,28 @@ For hardware-based recovery, the actual timer values programmed in hardware are The following diagram illustrates how the new `show pfcwd status` command retrieves and displays information: ```mermaid -flowchart LR - A[show pfcwd status] --> B[Query STATE_DB
PFC_WD_STATE Table] +flowchart TB + A[show pfcwd status] --> B[Query STATE_DB] + + B --> C[Read PFC_WD_GLOBAL] + B --> D[Read PFC_WD_STATE per port] - B --> C[Read Per-Port/Queue Entries] + C --> E[Extract mode: hardware/software] + C --> F[Extract detection_interval_range] + C --> G[Extract restoration_interval_range] - C --> D[Extract recovery_type] - C --> E[Extract detection_time_programmed] - C --> F[Extract detection_time_granularity] - C --> G[Extract restoration_time_programmed] - C --> H[Extract restoration_time_granularity] + D --> H[Extract programming_status] + D --> I[Extract detection_time_programmed] + D --> J[Extract restoration_time_programmed] - D --> L[Format Display Data] - E --> L - F --> L - G --> L - H --> L + E --> K[Format Display] + F --> K + G --> K + H --> K + I --> K + J --> K - L --> M[Display Table with:
PORT, RECOVERY TYPE,
HW DETECTION TIME, DETECTION GRANULARITY,
HW RESTORATION TIME, RESTORATION GRANULARITY] + K --> L[Display Output] ``` **Example: ASIC with Hardware Recovery** @@ -430,15 +436,23 @@ Now, let's use the new `show pfcwd status` command to see the actual hardware-pr ```shell admin@sonic:~$ show pfcwd status -PORT RECOVERY TYPE HW DETECTION TIME DETECTION GRANULARITY HW RESTORATION TIME RESTORATION GRANULARITY ----------- ------------- ------------------- --------------------- --------------------- ------------------------- -Ethernet0 hardware 300 100ms 500 100ms -Ethernet12 hardware 400 100ms 800 100ms +PFC Watchdog Mode: Hardware +Supported hardware detection interval range: 100-1500 ms +Supported hardware restoration interval range: 100-1000 ms + + PORT STATUS HW DETECTION TIME (ms) HW RESTORATION TIME (ms) +--------- ---------- ------------------------ -------------------------- +Ethernet0 success 400 400 +Ethernet8 success 800 800 +Ethernet320 failed N/A N/A ``` **Key Observations:** -- **Ethernet0**: Configuration (350ms/550ms) was adjusted to (300ms/500ms) by hardware due to 100ms granularity. The `show pfcwd config` shows what the user requested (350/550), but `show pfcwd status` reveals the actual hardware-programmed values (300/500). -- **Ethernet12**: Configuration (400ms/800ms) successfully programmed to hardware. Granularity is 100ms. +- **PFC Watchdog Mode**: Indicates whether hardware or software mode is in use +- **Supported ranges**: Displays the hardware capability ranges for detection and restoration timers (queried from SAI) +- **STATUS**: Shows programming status - "success" if hardware programming succeeded, "failed" if it failed +- **Hardware-programmed values**: The actual values programmed in hardware; shown as N/A if programming failed +- **Configuration verification**: The `show pfcwd config` shows what the user requested, while `show pfcwd status` reveals the actual hardware-programmed values and programming status **Note**: Since configuration validation is performed at both CLI and orchagent layers (see section 5.1), invalid configurations (e.g., values exceeding hardware timer range) are rejected before programming, ensuring all entries in `show pfcwd status` represent successfully programmed configurations. @@ -472,7 +486,6 @@ Following SAI statistics and attributes are used in this feature: | `SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` | sai_map_list_t | CREATE_AND_SET | Detection timer intervals per port/per TC in milliseconds | | `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL_RANGE` | sai_u32_range_t | READ_ONLY | Query supported recovery timer range per port (capability query) | | `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL` | sai_map_list_t | CREATE_AND_SET | Recovery timer intervals per port/per TC in milliseconds | -| `SAI_PORT_ATTR_PFC_TC_DLD_TIMER_INTERVAL` (Extension) | sai_map_list_t | CREATE_AND_SET | Port PFC Deadlock Detection timer granularity of all PFC priorities | **Switch Attributes (SAI_OBJECT_TYPE_SWITCH):** @@ -481,12 +494,13 @@ Following SAI statistics and attributes are used in this feature: | `SAI_SWITCH_ATTR_QUEUE_PFC_DEADLOCK_NOTIFY` | sai_pointer_t | CREATE_AND_SET | Register callback for PFC deadlock events | | `SAI_SWITCH_ATTR_PFC_DLR_PACKET_ACTION` | sai_packet_action_t | CREATE_AND_SET | Configure drop/forward action during recovery | | `SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE` | sai_u32_range_t | READ_ONLY | Query supported detection timer range at switch level (capability query) | -| `SAI_SWITCH_ATTR_PFC_TC_DLD_TIMER_INTERVAL` (Extension) | sai_map_list_t | CREATE_AND_SET | PFC Deadlock Detection timer granularity in milliseconds (1ms, 10ms, or 100ms) | +| `SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL` | sai_map_list_t | CREATE_AND_SET | Switch-level detection timer intervals per TC in milliseconds (global default for all ports) | +| `SAI_SWITCH_ATTR_PFC_TC_DLR_INTERVAL_RANGE` | sai_u32_range_t | READ_ONLY | Query supported recovery timer range at switch level (capability query) | +| `SAI_SWITCH_ATTR_PFC_TC_DLR_INTERVAL` | sai_map_list_t | CREATE_AND_SET | Switch-level recovery timer intervals per TC in milliseconds (global default for all ports) | **Notes:** -- Extension attributes are defined in `saiportextensions.h` and `saiswitchextensions.h` - The `*_RANGE` attributes are READ_ONLY and used to query hardware capabilities (min/max timer values) -- The `*_TIMER_INTERVAL` attributes are used to SET granularity values, not query them +- **Switch-level vs Port-level attributes**: Some platforms (e.g., Microsoft) use a single set of timers for all ports (switch-level). Other platforms may support per-port timer configuration. The switch-level attributes (`SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_SWITCH_ATTR_PFC_TC_DLR_INTERVAL`) can be used to set global defaults, which can be overridden by port-level attributes (`SAI_PORT_ATTR_PFC_TC_DLD_INTERVAL` / `SAI_PORT_ATTR_PFC_TC_DLR_INTERVAL`) when supported by the platform - There is no READ_ONLY attribute to query the current granularity setting; applications must track what they configured ### 6.2 SAI Events @@ -513,15 +527,28 @@ The hardware-based PFC watchdog uses the same configuration interface as the exi ### 7.1 STATE_DB Schema -The following STATE_DB table is used to track PFCWD detection and restoration events for both hardware and software-based methods. This table serves as the data source for the `show pfcwd status` command and telemetry reporting. +The following STATE_DB tables are used to track PFCWD global capabilities and per-port status for both hardware and software-based methods. These tables serve as the data source for the `show pfcwd status` command and telemetry reporting. + +**PFC_WD_GLOBAL Table:** + +This table tracks global PFC watchdog capabilities and mode. + +``` +PFC_WD_GLOBAL|GLOBAL + "mode": "hardware" | "software" + "detection_interval_range_min": // Hardware capability + "detection_interval_range_max": // Hardware capability + "restoration_interval_range_min": // Hardware capability + "restoration_interval_range_max": // Hardware capability +``` **PFC_WD_STATE Table:** -This table tracks the current state and configuration of PFC watchdog on each port/queue. +This table tracks the current state and programming status of PFC watchdog on each port/queue. ``` PFC_WD_STATE|| - "recovery_type": "hardware" | "software" + "programming_status": "success" | "failed" "status": "operational" | "storm_detected" | "storm_restored" "detection_count": "restoration_count": @@ -529,11 +556,9 @@ PFC_WD_STATE|| "last_restoration_time": "storm_duration_ms": "detection_time_configured": - "detection_time_programmed": // N/A for software mode - "detection_time_granularity": // N/A for software mode + "detection_time_programmed": // N/A for software mode or if programming failed "restoration_time_configured": - "restoration_time_programmed": // N/A for software mode - "restoration_time_granularity": // N/A for software mode + "restoration_time_programmed": // N/A for software mode or if programming failed "action": "drop" | "forward" ``` @@ -610,9 +635,11 @@ All existing PFC watchdog testing remains unchanged as documented in the [PFC Wa #### New CLI Command Testing - Verify `show pfcwd status` command displays all required fields correctly -- Test recovery type display (hardware vs software) -- Validate hardware timer values and granularity information display -- Test N/A display for software recovery ports +- Test PFC watchdog mode display (hardware vs software) +- Validate supported hardware interval ranges display +- Test programming status display (success vs failed) +- Validate hardware timer values display +- Test N/A display for failed programming status - Verify column alignment and formatting #### Configuration Validation Testing From 2ebf808ce046e8756d2f436daa6acb647b49f155 Mon Sep 17 00:00:00 2001 From: pinky-nexthop Date: Wed, 10 Jun 2026 14:07:32 +0530 Subject: [PATCH 15/15] HLD: decide PFCWD HW-vs-SW recovery path by SAI capability only Drive the PfcWdHwOrch-vs-PfcWdSwOrch path selection solely from the SAI capability query. Remove the SKU-based path determination and the operator override for path selection, so bringing a platform onto hardware recovery is purely a function of its SAI advertising the capability. Addresses review feedback from @eddyk-nvidia, @kperumalbfn, @yunang-c. Signed-off-by: pinky-nexthop --- ...fc_watchdog_hardware_detection_recovery.md | 84 ++----------------- 1 file changed, 5 insertions(+), 79 deletions(-) diff --git a/doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md b/doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md index 55e4a16dd68..debeb2b4f98 100644 --- a/doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md +++ b/doc/pfcwd/pfc_watchdog_hardware_detection_recovery.md @@ -18,7 +18,7 @@ - [Current Architecture](#current-architecture) - [Proposed Architecture](#proposed-architecture) - [Implementation Details](#implementation-details) - - [SKU-Based Override Mechanism](#sku-based-override-mechanism) + - [PfcWdHwOrch vs PfcWdSwOrch Selection](#pfcwdhworch-vs-pfcwdsworch-selection) - [5. CLI Changes](#5-cli-changes) - [5.1 Configuration Validation](#51-configuration-validation) - [5.2 New CLI command](#52-new-cli-command) @@ -46,6 +46,7 @@ |-----|------------|---------------|--------------------| | 0.1 | 12/13/2025 | Pinky Agrawal | Initial version | | 0.2 | 02/05/2026 | Pinky Agrawal | Addressed review comments: Added configuration validation using SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE, clarified counter polling mechanism, updated show pfcwd status for software mode, added telemetry test cases | +| 0.3 | 06/09/2026 | Pinky Agrawal | HW-vs-SW recovery path is now determined solely by the SAI capability query; removed the SKU-based determination and the operator override for path selection (addresses review feedback from @eddyk-nvidia, @kperumalbfn, @yunang-c) | ## Scope @@ -85,7 +86,7 @@ The hardware-based PFC watchdog recovery feature shall provide: 1. **Hardware Detection**: Automatic PFC deadlock detection in hardware without software polling 2. **Hardware Recovery**: Automatic recovery actions (drop/forward) performed by hardware 3. **Backward Compatibility**: Existing software-based PFC watchdog functionality remains unchanged -4. **Runtime Selection**: Automatic selection between hardware and software implementations based on platform capabilities +4. **Runtime Selection**: Automatic selection between hardware and software implementations based solely on the SAI capability query 5. **Enhanced CLI**: New `show pfcwd status` command to display recovery type, programming status, actual hardware timer values 6. **Error Handling**: Proper validation and error reporting for programming based on SAI errors @@ -245,84 +246,9 @@ Orch (base class) `PfcWdHwOrch` is the new class that will handle hardware recovery mechanisms. There will be no handlers defined in hardware-based recovery unlike software recovery which defines drop and forward handlers. To update the counters periodically, `PfcWdHwOrch` will use the same counter polling mechanism as software-based recovery. The counter querying remains the same as the software-based approach, using SAI queue statistics APIs (`SAI_QUEUE_STAT_PACKETS`, `SAI_QUEUE_STAT_DROPPED_PACKETS`, etc.) to read hardware counters and update COUNTERS_DB. -#### SKU-Based Override Mechanism - -In some cases, certain SKUs may need to use software-based PFCWD even when hardware capability is available. This can be due to: -- Hardware recovery in development mode -- Hardware limitations or bugs specific to certain SKU models -- Platform-specific requirements -- Testing or validation purposes - -For such cases, the orchagent can maintain a list of such SKUs and skip creating `PfcWdHwOrch` for those SKUs as given in the code below. - -Runtime selection in `orchdaemon.cpp`: - -```c -// Check hardware capability -bool pfcHwRecoverySupported = gSwitchOrch->checkPfcHwRecoverySupport(); - -// Check if SKU is in the override list to force software-based PFCWD -std::vector swPfcwdSkuList = getSwPfcwdSkuOverrideList(); -std::string currentSku = gSwitchOrch->querySwitchSku(); -bool skuForcesSwPfcwd = std::find(swPfcwdSkuList.begin(), swPfcwdSkuList.end(), currentSku) != swPfcwdSkuList.end(); - -// Check hardware capability -bool pfcHwRecoverySupported = gSwitchOrch->checkPfcHwRecoverySupport(); - -if (pfcHwRecoverySupported && !skuForcesSwPfcwd) -{ - SWSS_LOG_NOTICE("Starting hardware-based PFC watchdog (no handlers)"); - - // Query hardware timer range capabilities during initialization - sai_attribute_t attr_dld, attr_dlr; - attr_dld.id = SAI_SWITCH_ATTR_PFC_TC_DLD_INTERVAL_RANGE; - attr_dlr.id = SAI_SWITCH_ATTR_PFC_TC_DLR_INTERVAL_RANGE; - - sai_status_t status_dld = sai_switch_api->get_switch_attribute(gSwitchId, 1, &attr_dld); - sai_status_t status_dlr = sai_switch_api->get_switch_attribute(gSwitchId, 1, &attr_dlr); - - if (status_dld == SAI_STATUS_SUCCESS && status_dlr == SAI_STATUS_SUCCESS) - { - uint32_t detection_min = attr_dld.value.u32range.min; - uint32_t detection_max = attr_dld.value.u32range.max; - uint32_t restoration_min = attr_dlr.value.u32range.min; - uint32_t restoration_max = attr_dlr.value.u32range.max; - - SWSS_LOG_NOTICE("PFC watchdog hardware detection timer range: %u - %u ms", - detection_min, detection_max); - SWSS_LOG_NOTICE("PFC watchdog hardware restoration timer range: %u - %u ms", - restoration_min, restoration_max); - - // Store hardware capabilities in STATE_DB for CLI validation - DBConnector stateDb("STATE_DB", 0); - Table capabilitiesTable(&stateDb, "PFC_WD_HW_CAPABILITIES"); - - vector fvVector; - fvVector.emplace_back("detection_timer_min", to_string(detection_min)); - fvVector.emplace_back("detection_timer_max", to_string(detection_max)); - fvVector.emplace_back("restoration_timer_min", to_string(restoration_min)); - fvVector.emplace_back("restoration_timer_max", to_string(restoration_max)); - fvVector.emplace_back("recovery_type", "hardware"); - - capabilitiesTable.set("GLOBAL", fvVector); - SWSS_LOG_NOTICE("Stored PFC watchdog hardware capabilities in STATE_DB"); - } - else - { - SWSS_LOG_WARN("Failed to query PFC watchdog hardware timer ranges"); - } - - m_orchList.push_back(new PfcWdHwOrch( - m_configDb, - pfc_wd_tables, - portStatIds, - queueStatIds, - queueAttrIds)); -} -``` - - +#### PfcWdHwOrch vs PfcWdSwOrch Selection +`checkPfcHwRecoverySupport()` is called at init to decide whether to instantiate `PfcWdHwOrch` or fall back to `PfcWdSwOrch`. It returns true when the platform's SAI implementation advertises support for hardware-based PFC deadlock detection and recovery through a SAI capability query. ## 5. CLI Changes