[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow… by chartsai-nvidia · Pull Request #2385 · sonic-net/SONiC

chartsai-nvidia · 2026-06-11T22:56:44Z

Why I did it

Refines the SmartSwitch PMON HLD for DPU reboot-cause and midplane-down handling:

The old design assumed the NPU could read a DPU reboot-cause while the DPU was dead, triggered
only by an offline→online transition. Now the cause is captured only when the midplane is online,
using a per-boot boot_id so chassisd reliably detects a real DPU reboot.
Adds a get_midplane_down_reason() platform API and documents planned vs. unplanned midplane-down
reasons.

Work item tracking

Microsoft ADO (number only): N/A

How I did it

Reworked the DPU Reboot Cause flow around boot_id: the DPU publishes a fresh per-boot UUID into
CHASSIS_STATE_DB; the NPU chassisd compares it to the last persisted value and calls
get_reboot_cause() only on a real reboot with midplane up.
Added boot_id to the REBOOT_CAUSE and DPU_STATE schema examples.
Documented up→down midplane handling (planned via transition flag vs. unplanned via
get_midplane_down_reason()) and added the new API definition.

How to verify it

Repo	PR Title / Link	Status
sonic-platform-daemons	[chassisd] Capture DPU reboot cause via boot_id and add midplane-down reason for Smart Switch
sonic-host-services	[process-reboot-cause] Save boot_id and device fields for DPU reboot cause
sonic-platform-common	[ModuleBase] Add get_midplane_down_reason() API and MIDPLANE_DOWN_REASON_* constants
sonic-buildimage	[Mellanox] Implement get_midplane_down_reason for DPU module

…n_reason The commit updates 2 main parts: - when to run get_reboot_cause - get_midplane_down_reason Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-06-11T22:57:13Z

/azp run

azure-pipelines · 2026-06-11T22:57:19Z

No pipelines are associated with this pull request.

vvolam · 2026-06-23T16:52:19Z

+* Each DPU SONiC publishes a `boot_id` (a fresh UUID generated per boot from `/proc/sys/kernel/random/boot_id`) into its `DPU_STATE` entry in CHASSIS_STATE_DB. The NPU chassisd compares the reported `boot_id` against the last `boot_id` it persisted, when they differ a real DPU boot occurred, so chassisd calls `get_reboot_cause()` when midplane down transitions to up, records the cause to DB and json, and updates the persisted `boot_id`.
 * The get_reboot_cause will return the current reboot-cause of the module.
-* For persistent storage of the DPU reboot-cause and reboot-cause-history files use the existing mechanism and host storage path under "/host/reboot-cause/module/dpux".
+* For persistent storage of the DPU reboot-cause and reboot-cause-history files use the existing mechanism and host storage path under "/host/reboot-cause/module/dpux". The boot_id is also stored in this file.


Do you refer storing a boot_id file under the same dir?

The boot_id is stored in an additional field in the reboot-cause JSON.

liat-grozovik · 2026-07-15T12:30:16Z

@vvolam any further comments or this HLD can be approved and merged?
BTW you are welcome to review the code PRs as well as we wish to move forward with the merge process.

vvolam · 2026-07-15T17:39:51Z

@vvolam any further comments or this HLD can be approved and merged? BTW you are welcome to review the code PRs as well as we wish to move forward with the merge process.

@liat-grozovik In between the review. I will approve this week, if no further comments.

rameshraghupathy · 2026-07-15T20:44:31Z

 | 0.4 | 06/06/2024 | Ramesh Raghupathy | Added schema for DPU health-info and called out phase:1 and phase:2 activities for DPU health-info. Added key suffix to module reboot-cause to avoid key conflicts |
 | 0.5 | 04/30/2025 | Gagan Punathil Ellath | Added Post Startup and Pre shutdown sections for DPU |
 | 0.6 | 06/12/2025 | Gagan Punathil Ellath | DPU State Management Implementation |
+| 0.7 | 06/11/2026 | Charles Tsai | Updated DPU reboot-cause and midplane-down reason |


@chartsai-nvidia What is the rationale for this change?

This change updates the HLD to reflect the addition of the new reboot cause mechanism and get_midplane_down_reason() API. The version number is updated accordingly.

@chartsai-nvidia The question was why are we changing the existing behavior of capturing and reporting the reboot-cause of DPUs even when the DPUs are down, which was done intentionally in the existing implementation to help debug DPU failure.

mssonicbld · 2026-07-16T21:21:17Z

/azp run

azure-pipelines · 2026-07-16T21:21:24Z

No pipelines are associated with this pull request.

vvolam · 2026-07-17T21:42:02Z

@chartsai-nvidia Please fix DCO issue by amending your signoff to all commits.

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

mssonicbld · 2026-07-17T21:53:10Z

/azp run

azure-pipelines · 2026-07-17T21:53:16Z

No pipelines are associated with this pull request.

chartsai-nvidia · 2026-07-17T21:53:26Z

@chartsai-nvidia Please fix DCO issue by amending your signoff to all commits.

Done. Thanks

vvolam · 2026-07-20T21:57:38Z

+root@sonic:~#show system-health DPU DPU0
+
+Name        Oper-Status         State-Detail                    State-Value      Time                               Reason
+DPU0        Online              dpu_midplane_link_state         down             Wed Jun 10 06:39:57 PM UTC 2026    Unplanned: 'Thermal - Overload'


Just a nit, Oper-status will be offline when all the states are down.

vvolam

LGTM other than a small nit. @rameshraghupathy @seveerap79 do you have comments?

rameshraghupathy · 2026-07-23T14:49:32Z

 #### DPU Reboot Cause
 * The smartswitch needs to know the reboot cause for all the DPUs.
-* The NPU hardware should be capable of providing the DPU reboot-cause even when the DPUs are dead.
+* The reboot-cause (which is captured by the platform) is recorded only when the midplane of DPU is online by calling the `get_reboot_cause` API.


@chartsai-nvidia Can you phrase it clearly as shown? "The reboot-cause can be captured by the platforms at the appropriate time. The reboot-cause should be reported only when the midplane of DPU transitions to online by calling the get_reboot_cause API."

rameshraghupathy · 2026-07-23T15:06:14Z

 | 0.4 | 06/06/2024 | Ramesh Raghupathy | Added schema for DPU health-info and called out phase:1 and phase:2 activities for DPU health-info. Added key suffix to module reboot-cause to avoid key conflicts |
 | 0.5 | 04/30/2025 | Gagan Punathil Ellath | Added Post Startup and Pre shutdown sections for DPU |
 | 0.6 | 06/12/2025 | Gagan Punathil Ellath | DPU State Management Implementation |
+| 0.7 | 06/11/2026 | Charles Tsai | Updated DPU reboot-cause and midplane-down reason |


@chartsai-nvidia The question was why are we changing the existing behavior of capturing and reporting the reboot-cause of DPUs even when the DPUs are down, which was done intentionally in the existing implementation to help debug DPU failure.

rameshraghupathy · 2026-07-23T19:08:00Z


+#### Runtime DPU reboot-cause capture in chassisd (`boot_id` trigger)
+
+* A DPU reboot is detected when the NPU chassisd is notified of a change to the DPU's `boot_id` in CHASSIS_STATE_DB (via a DPU_STATE subscription). The reboot cause is fetched only when midplane up and a new `boot_id` is present. While the DPU is down or unreachable, there is no need to fetch a cause.


@chartsai-nvidia boot_id may change before midplane becomes reachable

rameshraghupathy · 2026-07-23T19:10:30Z

+
+* A DPU reboot is detected when the NPU chassisd is notified of a change to the DPU's `boot_id` in CHASSIS_STATE_DB (via a DPU_STATE subscription). The reboot cause is fetched only when midplane up and a new `boot_id` is present. While the DPU is down or unreachable, there is no need to fetch a cause.
+
+**DPU side (publisher).** The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE) when the boot_id is not in CHASSIS_STATE_DB.


@chartsai-nvidia Publisher must update when boot_id differs, not only when absent. Should this say that the DPU publishes the local boot_id when the field is absent or differs from the value currently present in CHASSIS_STATE_DB?

After a DPU reboot, the DB entry may still contain the previous boot ID. Checking only whether the field is absent would not update the DB with the new per-boot UUID.

rameshraghupathy · 2026-07-23T19:12:47Z

+
+**DPU side (publisher).** The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE) when the boot_id is not in CHASSIS_STATE_DB.
+
+**NPU side (consumer / capture).** The NPU chassisd subscribes to `DPU_STATE|DPUx` in CHASSIS_STATE_DB and reacts when `boot_id` changes. It keeps the last-captured boot_id per DPU in /host/reboot-cause/module/dpux/previous-reboot-cause.json. When the subscribed boot_id differs from the persisted value and midplane is up, the DPU has rebooted: chassisd calls `get_reboot_cause()`, writes boot_id and reboot cause to the file and CHASSIS_STATE_DB.


@chartsai-nvidia Please define the initialization and reconciliation behavior for the persisted and currently published boot IDs. At minimum, the HLD should cover:

persisted JSON missing, but the DB boot ID is present;

persisted JSON corrupt or missing the boot_id field;

persisted boot ID present, but DPU_STATE has not yet published a boot ID;

NPU/chassisd restart with the same DPU boot ID;

first upgrade to a software version that introduces this feature.

In particular, please clarify whether “no persisted boot ID + current boot ID present” records a reboot cause or only establishes the initial baseline.

rameshraghupathy · 2026-07-23T19:14:07Z

+
+**DPU side (publisher).** The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE) when the boot_id is not in CHASSIS_STATE_DB.
+
+**NPU side (consumer / capture).** The NPU chassisd subscribes to `DPU_STATE|DPUx` in CHASSIS_STATE_DB and reacts when `boot_id` changes. It keeps the last-captured boot_id per DPU in /host/reboot-cause/module/dpux/previous-reboot-cause.json. When the subscribed boot_id differs from the persisted value and midplane is up, the DPU has rebooted: chassisd calls `get_reboot_cause()`, writes boot_id and reboot cause to the file and CHASSIS_STATE_DB.


@chartsai-nvidia Please define the persistence ordering and retry behavior. The last-captured boot_id should not be advanced until the reboot-cause record has been successfully persisted.

For example, if previous-reboot-cause.json is updated first and the CHASSIS_STATE_DB/history write subsequently fails, chassisd could treat the boot ID as already processed and permanently suppress retry. The HLD should also require an atomic file update, such as temporary-file plus rename, to avoid leaving corrupt JSON after a crash.

rameshraghupathy · 2026-07-23T19:16:28Z

   * The switch updates the midplane state after querying the `is_midplane_reachable` platform API for the corresponding DPU. This is an universal implementation for all platforms
   * The midplane state is updated at a specific frequency by the chassisd running on the switch (The frequency is once every 10 seconds - as per `CHASSIS_INFO_UPDATE_PERIOD_SECS` in chassisd)
   * If the midplane state is down (which means that the DPU is no longer accessible through the midplane) This means that the state information which is present is no longer valid at the current instant, so the state information for control plane and data plane is set to 'down' but the data plane and the control plane reasons and the timestamps are retained as is for further debugging.
+   * On an `up -> down` midplane transition chassisd records **why** into `dpu_midplane_link_reason`:


@chartsai-nvidia Please define what constitutes an active transition. Checking only whether transition_type is set may allow a stale value from an earlier completed, timed-out, or failed operation to classify a later hardware failure as Planned.

Should this require state_transition_in_progress == true together with a valid transition_type and a non-expired transition_start_time? Please also document when these fields are cleared and what timeout is used.

rameshraghupathy · 2026-07-23T19:21:33Z

            A tuple (string, string) where the first element is a string containing the cause of the previous reboot. This string must be one of the predefined strings in this class. If the first string is "REBOOT_CAUSE_HARDWARE_OTHER", the second string can be used to pass a description of the reboot cause.
 ```

+#### Get midplane down reason


@chartsai-nvidia Why do we need a new API? The get_reboot_cause itself should be sufficient. Why midplane state alone needs an API?

[PMON HLD] update get_reboot_cause mechanism and add get_midplane_dow…

d37f07e

…n_reason The commit updates 2 main parts: - when to run get_reboot_cause - get_midplane_down_reason Signed-off-by: Charles Tsai <chartsai@nvidia.com>

gpunathilell requested review from rameshraghupathy and vvolam June 11, 2026 22:59

dgsudharsan previously approved these changes Jun 12, 2026

View reviewed changes

gpunathilell previously approved these changes Jun 22, 2026

View reviewed changes

vvolam reviewed Jun 23, 2026

View reviewed changes

rameshraghupathy reviewed Jul 15, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md Outdated

rameshraghupathy requested changes Jul 15, 2026

View reviewed changes

chartsai-nvidia dismissed stale reviews from gpunathilell and dgsudharsan via 95bf3bc July 16, 2026 21:21

Update smartswitch-pmon.md

761566a

Signed-off-by: Charles Tsai <chartsai@nvidia.com>

chartsai-nvidia force-pushed the chartsai/pmon-hld-update branch from 95bf3bc to 761566a Compare July 17, 2026 21:53

chartsai-nvidia requested a review from rameshraghupathy July 20, 2026 17:49

vvolam reviewed Jul 20, 2026

View reviewed changes

Comment thread doc/smart-switch/pmon/smartswitch-pmon.md

vvolam reviewed Jul 20, 2026

View reviewed changes

vvolam reviewed Jul 21, 2026

View reviewed changes

rameshraghupathy reviewed Jul 23, 2026

View reviewed changes

rameshraghupathy requested changes Jul 23, 2026

View reviewed changes


		#### Runtime DPU reboot-cause capture in chassisd (`boot_id` trigger)

		* A DPU reboot is detected when the NPU chassisd is notified of a change to the DPU's `boot_id` in CHASSIS_STATE_DB (via a DPU_STATE subscription). The reboot cause is fetched only when midplane up and a new `boot_id` is present. While the DPU is down or unreachable, there is no need to fetch a cause.


		* A DPU reboot is detected when the NPU chassisd is notified of a change to the DPU's `boot_id` in CHASSIS_STATE_DB (via a DPU_STATE subscription). The reboot cause is fetched only when midplane up and a new `boot_id` is present. While the DPU is down or unreachable, there is no need to fetch a cause.

		DPU side (publisher). The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE\|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE) when the boot_id is not in CHASSIS_STATE_DB.


		DPU side (publisher). The DPU `chassisd` reads the Linux per-boot UUID (`/proc/sys/kernel/random/boot_id`) and publishes it into its own `DPU_STATE\|DPUx` entry in the NPU's CHASSIS_STATE_DB (the same cross-DB write path DPUs already use for DPU_STATE) when the boot_id is not in CHASSIS_STATE_DB.

		NPU side (consumer / capture). The NPU chassisd subscribes to `DPU_STATE\|DPUx` in CHASSIS_STATE_DB and reacts when `boot_id` changes. It keeps the last-captured boot_id per DPU in /host/reboot-cause/module/dpux/previous-reboot-cause.json. When the subscribed boot_id differs from the persisted value and midplane is up, the DPU has rebooted: chassisd calls `get_reboot_cause()`, writes boot_id and reboot cause to the file and CHASSIS_STATE_DB.

Uh oh!

Conversation

chartsai-nvidia commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

Work item tracking

How I did it

How to verify it

Uh oh!

mssonicbld commented Jun 11, 2026

Uh oh!

azure-pipelines Bot commented Jun 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liat-grozovik commented Jul 15, 2026

Uh oh!

vvolam commented Jul 15, 2026

Uh oh!

Uh oh!

rameshraghupathy Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jul 16, 2026

Uh oh!

azure-pipelines Bot commented Jul 16, 2026

Uh oh!

vvolam commented Jul 17, 2026

Uh oh!

mssonicbld commented Jul 17, 2026

Uh oh!

azure-pipelines Bot commented Jul 17, 2026

Uh oh!

chartsai-nvidia commented Jul 17, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vvolam left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

chartsai-nvidia commented Jun 11, 2026 •

edited

Loading

rameshraghupathy Jul 15, 2026 •

edited

Loading