feat: silence detection – alert when jobs stop running entirely (v4.5.0) - #103
Merged
Conversation
Adds per-job opt-in silence detection. check-limits.php now uses CronExpression::getPreviousRunDate() to calculate the last expected start time and alerts (mail + Telegram) when no real execution has been recorded within the schedule interval plus a configurable grace period. Three maintenance-window guards prevent false positives: - Guard 1: agent-wide maintenance window → skip entirely - Guard 2: all job targets in maintenance → skip - Guard 3: most recent execution_log row was a maintenance sentinel (exit_code = -4) → skip (maintenance just ended) New job fields: notify_on_silence (bool, opt-in), silence_grace_minutes (int|null, per-job override), last_silence_alert_at (system-managed, reset by ExecutionStartEndpoint on next run, dedup: max 1 alert/hour). GET /health extended with silent_jobs (count) and last_execution_at (timestamp of most recent execution) for external monitoring. REST API (GET/POST/PUT /api/v1/jobs): notify_on_silence and silence_grace_minutes are readable and writable; last_silence_alert_at is read-only. Web UI: Silence-Alarm checkbox and optional grace-period field in the notifications tab; grace field shown/hidden via JS. DB migration: 016_silence_detection.sql adds three columns to cronjobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
check-limits.phpusesCronExpression::getPreviousRunDate()to detect when an active job has missed its scheduled start without leaving anyexecution_logentry. Alerts via Mail + Telegram.last_silence_alert_at); reset automatically when the job restarts.GET /healthextended withsilent_jobs(int|null) andlast_execution_at(string|null) for external monitors (UptimeKuma, Grafana).notify_on_silenceandsilence_grace_minutesreadable/writable on all job endpoints;last_silence_alert_atread-only.016_silence_detection.sqladds 3 columns tocronjobs.Affected files
agent/sql/migrations/016_silence_detection.sqlagent/bin/check-limits.phpagent/bin/send-notification.phptype = "silence"dispatch branchagent/src/Notification/MailNotifier.phpsendSilenceAlert()+ amber HTML templateagent/src/Notification/TelegramNotifier.phpsendSilenceAlert()agent/src/Endpoints/ExecutionStartEndpoint.phplast_silence_alert_at = NULLon job startagent/src/Endpoints/Cron{List,Get,Create,Update}Endpoint.phpagent/agent.php/healthextended with DB metricsweb/src/Controller/CronController.phpweb/templates/cron/form.phpweb/lang/de.php+en.phpAPI.mdagent/VERSION+web/VERSIONCHANGELOG.md+README.mdTest plan
016_silence_detection.sqlon target DBnotify_on_silence = 1, let it become overdue → silence alert received via Mail + Telegramlast_silence_alert_at = NULL, no further alertGET /healthshowssilent_jobs: 1while overdue,0after recoverysilence_grace_minutesper job overrides globalsilence.grace_minutesnotify_on_silenceandsilence_grace_minutesround-trip correctly