Skip to content

[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow… #2385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chartsai-nvidia wants to merge 2 commits into
base: master
Choose a base branch
from
+35 −8
Open

[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow… #2385

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d37f07e
[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow…
chartsai-nvidia Jun 11, 2026
761566a
Update smartswitch-pmon.md
chartsai-nvidia Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 35 additions & 8 deletions doc/smart-switch/pmon/smartswitch-pmon.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
| 0.4 | 06/06/2024 | Ramesh Raghupathy | Added schema for DPU health-info and called out phase:1 and phase:2 activities for DPU health-info. Added key suffix to module reboot-cause to avoid key conflicts |
| 0.5 | 04/30/2025 | Gagan Punathil Ellath | Added Post Startup and Pre shutdown sections for DPU |
| 0.6 | 06/12/2025 | Gagan Punathil Ellath | DPU State Management Implementation |
| 0.7 | 06/11/2026 | Charles Tsai | Updated DPU reboot-cause and midplane-down reason |

@rameshraghupathy rameshraghupathy Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia What is the rationale for this change?

@chartsai-nvidia chartsai-nvidia Jul 20, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change updates the HLD to reflect the addition of the new reboot cause mechanism and get_midplane_down_reason() API. The version number is updated accordingly.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia The question was why are we changing the existing behavior of capturing and reporting the reboot-cause of DPUs even when the DPUs are down, which was done intentionally in the existing implementation to help debug DPU failure.

@gpunathilell gpunathilell Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rameshraghupathy This was already discussed in the platform meeting, we need to make sure that the behavior is aligned to the switch side, we do not want 2 reboot causes for each DPU reboots (->offline->Online) . That is why the get_midplane_down_reason() is being added newly so that the platform can still provide additional info to help debug DPU failure. Do you want this to be explicitly recorded?


## Definitions / Abbreviations

Expand Down Expand Up @@ -614,9 +615,10 @@ The smartswitch needs to know the reboot cause for the NPU and the DPUs.
```
#### DPU Reboot Cause
* The smartswitch needs to know the reboot cause for all the DPUs.
* The NPU hardware should be capable of providing the DPU reboot-cause even when the DPUs are dead.
* The reboot-cause (which is captured by the platform) is recorded only when the midplane of DPU is online by calling the `get_reboot_cause` API.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Can you phrase it clearly as shown? "The reboot-cause can be captured by the platforms at the appropriate time. The reboot-cause should be reported only when the midplane of DPU transitions to online by calling the get_reboot_cause API."

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Will change into "The reboot-cause can be captured by the platforms at the appropriate time. The reboot-cause should be reported only when the midplane of DPU transitions to online by calling the get_reboot_cause API."

* Each DPU SONiC publishes a `boot_id` (a fresh UUID generated per boot from `/proc/sys/kernel/random/boot_id`) into its `DPU_STATE` entry in CHASSIS_STATE_DB. The NPU chassisd compares the reported `boot_id` against the last `boot_id` it persisted, when they differ a real DPU boot occurred, so chassisd calls `get_reboot_cause()` when midplane down transitions to up, records the cause to DB and json, and updates the persisted `boot_id`.
* The get_reboot_cause will return the current reboot-cause of the module.
* For persistent storage of the DPU reboot-cause and reboot-cause-history files use the existing mechanism and host storage path under "/host/reboot-cause/module/dpux".
* For persistent storage of the DPU reboot-cause and reboot-cause-history files use the existing mechanism and host storage path under "/host/reboot-cause/module/dpux". The boot_id is also stored in this file.

@vvolam vvolam Jun 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you refer storing a boot_id file under the same dir?

@chartsai-nvidia chartsai-nvidia Jun 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The boot_id is stored in an additional field in the reboot-cause JSON.

* The storage and retrieval of the reboot-cause of the Switch and PDUs are shown in the sequence diagram

<p align="center"><img src="./images/dpu-reboot-seq.svg"></p>
Expand All @@ -628,9 +630,6 @@ The smartswitch needs to know the reboot cause for the NPU and the DPUs.
* Updates the NPU reboot-cause into the StateDB and the DPU reboot-cause into the ChassisStateDB.
* The above process is a one-shot event on boot up.
* The module_db_update function in the NPU-PMON chassisd is an existing function constantly updating the operational status of the DPUs.
* This function looks for DPU operational status change events and when the DPUs come out of "offline" state, issues "get_reboot_cause" API to the platform.
* The dpu operational state transition from 'offline' to 'online' guarantees the reboot of a DPU.
* The platform code will extract the DPU reboot cause from the NPU hardware itself even when the DPU is not reachable.
* The DPU reboot cause will be mapped to one of the following existing reboot causes and returned back.
* REBOOT_CAUSE_POWER_LOSS = "Power Loss"
* REBOOT_CAUSE_THERMAL_OVERLOAD_CPU = "Thermal Overload: CPU"
Expand All @@ -651,12 +650,21 @@ return REBOOT_CAUSE_NON_HARDWARE + ', ' + 'kernel panic'
* The switch reboot use case will follow the same sequence.
* The pmon container restart should not affect this sequence as the states are persisted either in the DB or in the file system.

#### Runtime DPU reboot-cause capture in chassisd (`boot_id` trigger)

* A DPU reboot is detected when the NPU chassisd is notified of a change to the DPU's `boot_id` in CHASSIS_STATE_DB (via a DPU_STATE subscription). The reboot cause is fetched only when midplane up and a new `boot_id` is present. While the DPU is down or unreachable, there is no need to fetch a cause.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia boot_id may change before midplane becomes reachable

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boot_id may change before midplane becomes reachable, but get_reboot_cause can only be call from NPU, this is a known limitation. There is no need to record boot_id if we cannot get reboot cause.


**DPU side (publisher).** The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE) when the boot_id is not in CHASSIS_STATE_DB.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Publisher must update when boot_id differs, not only when absent. Should this say that the DPU publishes the local boot_id when the field is absent or differs from the value currently present in CHASSIS_STATE_DB?

After a DPU reboot, the DB entry may still contain the previous boot ID. Checking only whether the field is absent would not update the DB with the new per-boot UUID.

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, will check


**NPU side (consumer / capture).** The NPU chassisd subscribes to `DPU_STATE|DPUx` in CHASSIS_STATE_DB and reacts when `boot_id` changes. It keeps the last-captured boot_id per DPU in /host/reboot-cause/module/dpux/previous-reboot-cause.json. When the subscribed boot_id differs from the persisted value and midplane is up, the DPU has rebooted: chassisd calls `get_reboot_cause()`, writes boot_id and reboot cause to the file and CHASSIS_STATE_DB.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Please define the initialization and reconciliation behavior for the persisted and currently published boot IDs. At minimum, the HLD should cover:

  • persisted JSON missing, but the DB boot ID is present;
  • persisted JSON corrupt or missing the boot_id field;
  • persisted boot ID present, but DPU_STATE has not yet published a boot ID;
  • NPU/chassisd restart with the same DPU boot ID;
  • first upgrade to a software version that introduces this feature.

In particular, please clarify whether “no persisted boot ID + current boot ID present” records a reboot cause or only establishes the initial baseline.

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Understood. Will update.

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Please define the persistence ordering and retry behavior. The last-captured boot_id should not be advanced until the reboot-cause record has been successfully persisted.

For example, if previous-reboot-cause.json is updated first and the CHASSIS_STATE_DB/history write subsequently fails, chassisd could treat the boot ID as already processed and permanently suppress retry. The HLD should also require an atomic file update, such as temporary-file plus rename, to avoid leaving corrupt JSON after a crash.

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, will take a look


#### Schema for REBOOT_CAUSE of DPUs on switch ChassisStateDB
```
Key: "REBOOT_CAUSE|DPU0|2024_06_06_09_31_18"

"REBOOT_CAUSE|DPU0|2024_06_06_09_31_18": {
"value": {
"boot_id": <boot-id>,
"cause": "Software causes (Reboot)",
"comment": "User issued 'reboot' command [User: admin, Time: Thu Jun 6 09:46:43 AM UTC 2024]",
"device": "DPU0",
Expand Down Expand Up @@ -692,6 +700,7 @@ dpu_data_plane_state: up refers to configuration downloaded, the pipeline stage
key: DPU_STATE|DPU0

"id": "1",
"boot_id": <boot-id>,
Comment thread
chartsai-nvidia marked this conversation as resolved.
”dpu_midplane_link_state”: "up"
“dpu_midplane_link_time": "timestamp",
"dpu_midplane_link_reason": "up_down_related string",
Expand All @@ -710,6 +719,8 @@ The DPU state management is implemented through a combination of classes that ha
* The switch updates the midplane state after querying the `is_midplane_reachable` platform API for the corresponding DPU. This is an universal implementation for all platforms
* The midplane state is updated at a specific frequency by the chassisd running on the switch (The frequency is once every 10 seconds - as per `CHASSIS_INFO_UPDATE_PERIOD_SECS` in chassisd)
* If the midplane state is down (which means that the DPU is no longer accessible through the midplane) This means that the state information which is present is no longer valid at the current instant, so the state information for control plane and data plane is set to 'down' but the data plane and the control plane reasons and the timestamps are retained as is for further debugging.
* On an `up -> down` midplane transition chassisd records **why** into `dpu_midplane_link_reason`:

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Please define what constitutes an active transition. Checking only whether transition_type is set may allow a stale value from an earlier completed, timed-out, or failed operation to classify a later hardware failure as Planned.

Should this require state_transition_in_progress == true together with a valid transition_type and a non-expired transition_start_time? Please also document when these fields are cleared and what timeout is used.

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I'll take further look and fix it.

If the transition flag (`transition_type`) is set, it is **Planned** down and we record the transition flag. If the flag is not set, it's **Unplanned** shutdown and we record the result from `get_midplane_down_reason()` platform-dependent API and save in CHASSIS_STATE_DB.

2. **DpuStateManagerTask Class - present in Chassisd on DPU - Dependent on chassisd being enabled on DPU**
* If there is a platform specific implementation of `get_dataplane_state` and `get_controlplane_state` implementation, Then these functions are called in polling mode by the chassisd running on DPU to update the relevant state information
Expand Down Expand Up @@ -790,6 +801,15 @@ def get_reboot_cause(self):
A tuple (string, string) where the first element is a string containing the cause of the previous reboot. This string must be one of the predefined strings in this class. If the first string is "REBOOT_CAUSE_HARDWARE_OTHER", the second string can be used to pass a description of the reboot cause.
```

#### Get midplane down reason

@rameshraghupathy rameshraghupathy Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chartsai-nvidia Why do we need a new API? The get_reboot_cause itself should be sufficient. Why midplane state alone needs an API?

def get_midplane_down_reason(self):
```
Retrieves the cause of the midplane down reason

Returns:
A tuple (string, string) where the first element is a string containing the cause of midplane down reason. This string must be one of the predefined strings in this class. If the first string is "MIDPLANE_DOWN_REASON_HARDWARE_OTHER", the second string can be used to pass a description of the midplane down reason.
```

#### DPU_STATE Use Case
* This API can be be used by High Availability, Load Balancing, Debug, error recovery (reset, power cycle) and fault management logics. The exact HA use case will be updated in the HA document.

Expand Down Expand Up @@ -1045,7 +1065,7 @@ System status summary
* The "show system-health ..." is extended to include the DPU_STATE detail.
show system-health DPU \<dpu-index\> <font>**`Executed on the switch. This CLI is not available on the DPU.`**</font>
```
When the idex is "all" shows the detailed state of all DPUs
When the index is "all" shows the detailed state of all DPUs

Oper-Status definition:
Online : All states are up
Expand All @@ -1065,12 +1085,19 @@ DPU1 Online dpu_midplane_link_state up W
dpu_control_plane_state up Wed 20 Oct 2023 06:52:28 PM UTC
dpu_data_plane_state up Wed 20 Oct 2023 06:52:28 PM UTC

root@sonic:~#show system-health DPU 0
root@sonic:~#show system-health DPU DPU0

Name Oper-Status State-Detail State-Value Time Reason
DPU0 Offline dpu_midplane_link_state down Wed 20 Oct 2023 06:52:28 PM UTC PCIe link is down
DPU0 Offline dpu_midplane_link_state down Wed 20 Oct 2023 06:52:28 PM UTC Planned: 'shutdown'
dpu_control_plane_state down Wed 20 Oct 2023 06:52:28 PM UTC
dpu_data_plane_state down Wed 20 Oct 2023 06:52:28 PM UTC

root@sonic:~#show system-health DPU DPU0

Name Oper-Status State-Detail State-Value Time Reason
DPU0 Online dpu_midplane_link_state down Wed Jun 10 06:39:57 PM UTC 2026 Unplanned: 'Thermal - Overload'

@vvolam vvolam Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a nit, Oper-status will be offline when all the states are down.

@chartsai-nvidia chartsai-nvidia Jul 23, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Will fix.

dpu_control_plane_state down Thu Jun 11 05:25:49 PM UTC 2026
dpu_data_plane_state down Thu Jun 11 05:25:49 PM UTC 2026
```
#### System health cli extended further as shown
* Detailed output from the switch can be obtained with the following CLI
Expand Down