alarmd — Alarm Framework for SONiC [HLD]#2314
Conversation
Signed-off-by: Neel Datta <neel.datta@hpe.com>
Signed-off-by: Neel Datta <neel.datta@hpe.com>
|
/azp run |
|
No pipelines are associated with this pull request. |
| - **System resources**: CPU, memory, disk usage (similar to monit, but | ||
| integrated into SYSTEM_ALARMS) | ||
| - **Container health**: Check if critical Docker containers are running | ||
| - **Platform-specific hardware**: FPGA status, I2C bus health, GPIO state, |
There was a problem hiding this comment.
Does it have to be either a bash or python script? Could it be a binary?
There was a problem hiding this comment.
Will look into testing this, if feasible I'll update HLD accordingly; I may create a separate design doc for the script runner itself.
| ### 5.1 Functional Requirements | ||
|
|
||
| - alarmd shall poll STATE_DB tables at a configurable interval (default 3 seconds) and evaluate field conditions defined in alarm definition files. | ||
| - alarmd shall execute external health-check scripts at a configurable interval (default 60 seconds) and evaluate exit codes. |
There was a problem hiding this comment.
Does it have to be a script that runs on a per-interval basis or could I run a health check script just once? For instance, what if I want to check the status of a platform init using a script during boot? I don't suppose there would be a need to run that script repeatedly, would there?
|
There already exists a relatively similar daemon called
Have you considered whether this daemon could be extended to aggregate and report hardware alarms as opposed to introducing a new daemon? |
| ### 13.2 System Tests | ||
|
|
||
| Functional tests run on a live DUT over SSH, injecting faults via direct | ||
| STATE_DB writes and verifying `SYSTEM_ALARMS` response. | ||
|
|
||
| | Category | Scenarios | | ||
| |----------|-----------| | ||
| | Alarm raise / clear | PSU missing, fan under-speed, thermal warning — verify raise within 2 poll intervals, clear on recovery | | ||
| | Multi-FRU independence | Fault on PSU 1 does not affect PSU 2 | | ||
| | OR-logic | Voltage low / high both raise `PSU_OUTPUT_VOLTAGE_FAULT`; clears only when in range | | ||
| | Script checks | Inject fault script → alarm raised; restore → cleared | | ||
| | SIGHUP reload | Valid reload → new thresholds take effect; invalid JSON → old config preserved | | ||
| | Daemon restart | Stop/start with active fault → alarms cleared then re-raised within one poll | | ||
| | Warmboot / fastboot | `clear_on_startup()` removes stale alarms; faults re-raised; no boot delay | | ||
| | CLI | `show alarms`, `show alarms --summary`, `show alarms --json`, filter options (`-s`, `-c`, `-o`, `-g`) output verification | | ||
|
|
There was a problem hiding this comment.
sonic-mgmtify these tests
|
|
||
| ### 14.2 Alarm History Table | ||
|
|
||
| **Motivation**: Operators and NMS/gNMI collectors may want to query historical |
There was a problem hiding this comment.
If we stream the SYSTEM_ALARMS table out, wouldn't collectors have per-switch alarm history anyway?
|
/azp run |
|
No pipelines are associated with this pull request. |
|
/azp run |
|
No pipelines are associated with this pull request. |
Signed-off-by: Neel Datta <neel.datta@hpe.com>
|
/azp run |
|
No pipelines are associated with this pull request. |
|
@neelDatta17 Can we align to the existing Alarm design in the community? |
venkatmahalingam
left a comment
venkatmahalingam
left a comment
There was a problem hiding this comment.
New design should be aligned to the existing alarm design.
|
/azp run |
|
No pipelines are associated with this pull request. |
|
/azp run |
|
No pipelines are associated with this pull request. |
|
/azp run |
|
No pipelines are associated with this pull request. |
Rev 0.2. alarmd publishes RAISE/CLEAR through eventd's event_publish() API and relies on eventd's ALARM/ALARM_STATS tables (EVENT_DB, Redis DB 19) instead of a private SYSTEM_ALARMS table. Design verified against the merged eventd source (eventutils.h, eventconsume.cpp): bare type-id|resource alarm identity, severity assigned by eventd from the event profile, unprofiled type-ids dropped. Adds a forward-looking event-subscription adapter (STATE_DB polling is the working v1 path), periodic reconciliation against the ALARM table, event-profile/YANG registration, and systemd ordering after eventd. Diagrams converted to Mermaid; removed the superseded PNGs. Signed-off-by: Neel Datta <neel.datta@hpe.com>
|
/azp run |
|
No pipelines are associated with this pull request. |
|
@bhaveshdell Will review this PR from Dell. |
5 participants
The high level design document for alarmd, a new daemon that provides centralized, structured hardware alarm detection for SONiC. It consumes existing STATE_DB data (zero changes to existing daemons), evaluates fault conditions defined entirely in JSON config, and writes unified alarm state to a SYSTEM_ALARMS table with severity, category, and lifecycle tracking.