diff --git a/API.md b/API.md index 917cf19..4123b5a 100644 --- a/API.md +++ b/API.md @@ -316,6 +316,9 @@ List all cron jobs. "active": true, "notify_on_failure": true, "notify_on_recovery": false, + "notify_on_silence": false, + "silence_grace_minutes": null, + "last_silence_alert_at": null, "execution_limit_seconds": 300, "auto_kill_on_limit": false, "singleton": false, @@ -372,6 +375,8 @@ Create a new cron job. Scope: **`jobs:write`** "tags": ["backup"], "notify_on_failure": true, "notify_on_recovery": false, + "notify_on_silence": false, + "silence_grace_minutes": null, "execution_limit_seconds": 0, "auto_kill_on_limit": false, "singleton": false, @@ -951,6 +956,7 @@ to 60 seconds of delay before the daemon picks up the entry. | Version | Change | |---|---| +| 4.5.0 | Added `notify_on_silence` (bool), `silence_grace_minutes` (int\|null), `last_silence_alert_at` (string\|null, read-only) to job objects; `GET /health` extended with `silent_jobs` (int\|null) and `last_execution_at` (string\|null) | | 4.3.4 | Added `GET /api/v1/audit` endpoint (`audit:read` scope, admin-only); added §15 Audit Log | | 4.2.0 | Added `GET /api/v1/agents` endpoint (`settings:read` scope; respects `agent_ids` restriction; omits sensitive fields) | | 4.1.0 | Initial external REST API with API key authentication and scope-based authorization | diff --git a/CHANGELOG.md b/CHANGELOG.md index 4295f63..9817bc9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,39 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). --- +## [4.5.0] – branch: `feature/silence-detection` + +### Added + +- **Silence Detection**: Aktive Jobs mit `notify_on_silence = 1` werden von `check-limits.php` + minütlich überwacht. Wenn ein Job seinen letzten geplanten Startzeitpunkt (berechnet via + `CronExpression::getPreviousRunDate()`) plus Toleranzzeit überschritten hat, ohne einen echten + Start (exit_code ≠ -4) zu verzeichnen, wird eine Benachrichtigung per Mail und/oder Telegram + ausgelöst. Erkennt stumme Komplettausfälle, die keine `execution_log`-Einträge erzeugen. +- **Drei Maintenance-Guards** gegen False Positives: + - Guard 1: Agent-weite Maintenance → Silence-Detection übersprungen. + - Guard 2: Alle Targets des Jobs in Maintenance → kein Alert. + - Guard 3: Letzter `execution_log`-Eintrag war ein Maintenance-Sentinel (exit_code = -4) → + kein Alert (Maintenance gerade erst beendet). +- **Dedup**: Alert wird höchstens einmal pro Stunde pro Job gesendet (`last_silence_alert_at`). + Das Feld wird in `ExecutionStartEndpoint` automatisch auf `NULL` zurückgesetzt, wenn der Job + wieder startet. +- **Neue Job-Felder**: `notify_on_silence` (bool, opt-in, Default 0), `silence_grace_minutes` + (int|null, überschreibt globalen `silence.grace_minutes`-Wert aus `config.json`), + `last_silence_alert_at` (datetime|null, system-only, read-only über API). +- **MailNotifier::sendSilenceAlert()** und **TelegramNotifier::sendSilenceAlert()**: neue Methoden + mit amber/gelber Farbgebung (Mail-Template) und 🔇-Emoji (Telegram). +- **`send-notification.php`**: neuer Zweig `type = "silence"` für asynchronen Dispatch. +- **`GET /health`** erweitert um `silent_jobs` (Anzahl aktiver Jobs mit überschrittenem + Silence-Schwellwert, `null` bei DB-Fehler) und `last_execution_at` (letzter `started_at`-Wert + aus `execution_log`, ermöglicht externe Uptime-Monitore ohne Per-Job-Konfiguration). +- **REST API**: `notify_on_silence` und `silence_grace_minutes` in GET/POST/PUT für Jobs; + `last_silence_alert_at` ist read-only. +- **Web-Formular**: Checkbox „Silence-Alarm aktivieren" und optional Toleranzzeit-Feld im + Benachrichtigungs-Tab; Grace-Feld wird per JS ein-/ausgeblendet. + +--- + ## [4.4.4] – branch: `fix/hmac-agent-username-default` ### Fixed diff --git a/README.md b/README.md index 956f223..4229955 100644 --- a/README.md +++ b/README.md @@ -70,6 +70,7 @@ history, email failure alerts, execution limits, multi-host support, and SSO int | **Email alerts** | Receive an email when a job exits with a non-zero status or exceeds its execution limit | | **Telegram alerts** | Receive a Telegram message for the same events via the Bot API | | **Recovery notifications** | Optionally receive an email and/or Telegram message when a job succeeds again after a failure streak that triggered an alert | +| **Silence detection** | Opt-in per job: `check-limits.php` uses the cron schedule to calculate the last expected start time and alerts (email + Telegram) if no real execution has been recorded within the schedule interval plus a configurable grace period. Three maintenance-window guards prevent false positives. `GET /health` exposes a `silent_jobs` counter for external monitors | | **Maintenance Windows** | Define per-target scheduled maintenance windows; jobs are either skipped (exit code −4) or executed silently depending on the per-job setting. A special **"Cronmanager Agent"** target blocks all executions host-wide (useful for VM maintenance cycles). Conflict icons (⚠ amber / ✕ red) appear in the job list and detail view | | **SSH connectivity test** | A **Test** button on the Maintenance Windows page verifies that the agent can reach an SSH target via key-based auth (`BatchMode=yes`, 10 s timeout). The result (Connected / Failed) is shown inline without a page reload | | **Startup orphan cleanup** | On agent restart, executions still marked as "running" with no live process are automatically resolved to exit code −5 ("Interrupted by system restart") | diff --git a/agent/VERSION b/agent/VERSION index cbe06cd..a84947d 100644 --- a/agent/VERSION +++ b/agent/VERSION @@ -1 +1 @@ -4.4.4 +4.5.0 diff --git a/agent/agent.php b/agent/agent.php index 9eb8fc2..e59ed9e 100644 --- a/agent/agent.php +++ b/agent/agent.php @@ -171,16 +171,50 @@ function jsonResponse(int $statusCode, array $data): void $router = new Router(); - // -- Health (fully functional, HMAC-exempt) -------------------------------- + // -- Crons ---------------------------------------------------------------- + + // -- Health (HMAC-exempt, registered after PDO init to include DB metrics) - $router->addRoute('GET', '/health', function (array $params) use ($logger): void { $logger->info('Health check requested'); $versionFile = __DIR__ . '/VERSION'; + + $silentJobs = null; + $lastExecutionAt = null; + + try { + $healthPdo = \Cronmanager\Agent\Database\Connection::getInstance()->getPdo(); + + $graceMinutes = 10; // same default used by check-limits.php silence detection + + $silentStmt = $healthPdo->query( + "SELECT COUNT(*) FROM cronjobs c + WHERE c.active = 1 + AND c.notify_on_silence = 1 + AND ( + SELECT MAX(CASE WHEN e.exit_code != -4 THEN e.started_at END) + FROM execution_log e + WHERE e.cronjob_id = c.id + ) < DATE_SUB(NOW(), INTERVAL COALESCE(c.silence_grace_minutes, {$graceMinutes}) MINUTE)" + ); + $silentJobs = (int) $silentStmt->fetchColumn(); + + $lastExecStmt = $healthPdo->query( + 'SELECT MAX(started_at) FROM execution_log' + ); + $raw = $lastExecStmt->fetchColumn(); + $lastExecutionAt = ($raw !== false && $raw !== null) ? (string) $raw : null; + } catch (\Throwable $e) { + $logger->warning('Health: DB query failed', ['message' => $e->getMessage()]); + } + jsonResponse(200, [ 'status' => 'ok', 'version' => is_readable($versionFile) ? trim((string) file_get_contents($versionFile)) : 'unknown', 'container_version' => getenv('APP_VERSION') ?: 'unknown', 'timestamp' => date('c'), + 'silent_jobs' => $silentJobs, + 'last_execution_at' => $lastExecutionAt, ]); }); diff --git a/agent/bin/check-limits.php b/agent/bin/check-limits.php index 110d325..7b3c138 100644 --- a/agent/bin/check-limits.php +++ b/agent/bin/check-limits.php @@ -54,10 +54,12 @@ } }); +use Cron\CronExpression; use Cronmanager\Agent\Bootstrap; use Cronmanager\Agent\Database\Connection; use Cronmanager\Agent\Notification\MailNotifier; use Cronmanager\Agent\Notification\TelegramNotifier; +use Cronmanager\Agent\Repository\MaintenanceWindowRepository; // --------------------------------------------------------------------------- // Bootstrap @@ -120,19 +122,18 @@ exit(1); } -if (empty($exceeded)) { - $logger->debug('check-limits: no executions exceeding their limit'); - exit(0); -} - -$logger->info('check-limits: found executions exceeding limit', ['count' => count($exceeded)]); - $mailNotifier = new MailNotifier($logger, $config); $telegramNotifier = new TelegramNotifier($logger, $config); -$notifyScript = __DIR__ . '/send-notification.php'; -$execAvailable = function_exists('exec') +$notifyScript = __DIR__ . '/send-notification.php'; +$execAvailable = function_exists('exec') && !in_array('exec', array_map('trim', explode(',', (string) ini_get('disable_functions'))), true); +if (empty($exceeded)) { + $logger->debug('check-limits: no executions exceeding their limit'); +} else { + $logger->info('check-limits: found executions exceeding limit', ['count' => count($exceeded)]); +} + foreach ($exceeded as $row) { $executionId = (int) $row['execution_id']; $jobId = (int) $row['job_id']; @@ -342,6 +343,233 @@ } } +// --------------------------------------------------------------------------- +// Silence Detection +// Checks jobs with notify_on_silence = 1 and alerts when no real start +// (exit_code != -4) was recorded within the expected schedule + grace period. +// --------------------------------------------------------------------------- + +$logger->debug('check-limits: starting silence detection'); + +$maintenanceRepo = new MaintenanceWindowRepository($pdo, $logger); +$globalGraceMinutes = max(1, (int) $config->get('silence.grace_minutes', 10)); + +// Guard 1: Agent-wide maintenance window active → skip entirely +if ($maintenanceRepo->isAgentInMaintenance()) { + $logger->info('check-limits: agent in maintenance, skipping silence detection'); +} else { + try { + $silenceStmt = $pdo->query( + "SELECT + c.id, + c.description, + c.schedule, + c.linux_user, + c.execution_mode, + c.ssh_host, + c.execution_limit_seconds, + c.silence_grace_minutes, + c.last_silence_alert_at, + c.created_at, + GROUP_CONCAT(DISTINCT jt.target ORDER BY jt.target SEPARATOR ',') AS targets, + MAX(e.started_at) AS last_any_start, + MAX(CASE WHEN e.exit_code != -4 THEN e.started_at END) AS last_real_start + FROM cronjobs c + LEFT JOIN execution_log e ON e.cronjob_id = c.id + LEFT JOIN job_targets jt ON jt.job_id = c.id + WHERE c.active = 1 + AND c.notify_on_silence = 1 + GROUP BY c.id" + ); + $silenceRows = $silenceStmt->fetchAll(); + } catch (\Throwable $e) { + $logger->error('check-limits: failed to query silence-detection jobs', ['message' => $e->getMessage()]); + $silenceRows = []; + } + + $tzName = date_default_timezone_get(); + $tz = new \DateTimeZone($tzName); + + foreach ($silenceRows as $srow) { + $jobId = (int) $srow['id']; + $description = (string) ($srow['description'] ?? ''); + $schedule = (string) $srow['schedule']; + $createdAt = (string) $srow['created_at']; + $lastRealStart = $srow['last_real_start'] !== null ? (string) $srow['last_real_start'] : null; + $lastAnyStart = $srow['last_any_start'] !== null ? (string) $srow['last_any_start'] : null; + $lastAlertAt = $srow['last_silence_alert_at'] !== null ? (string) $srow['last_silence_alert_at'] : null; + + // Derive targets list (same fallback logic as CronListEndpoint) + $targetsRaw = $srow['targets'] !== null ? (string) $srow['targets'] : ''; + if ($targetsRaw !== '') { + $targets = explode(',', $targetsRaw); + } else { + $mode = (string) ($srow['execution_mode'] ?? 'local'); + $sshHost = isset($srow['ssh_host']) ? trim((string) $srow['ssh_host']) : ''; + $targets = ($mode === 'remote' && $sshHost !== '') ? [$sshHost] : ['local']; + } + + // Guard 2: All targets in maintenance → skip + $allInMaintenance = true; + foreach ($targets as $tgt) { + if (!$maintenanceRepo->isTargetInMaintenance($tgt)) { + $allInMaintenance = false; + break; + } + } + if ($allInMaintenance) { + $logger->debug('check-limits: silence skipped (all targets in maintenance)', ['job_id' => $jobId]); + continue; + } + + // Guard 3: Most recent execution_log row was a maintenance sentinel → maintenance just ended + // Detected when last_any_start is newer than last_real_start (or real is null while any is not) + if ($lastAnyStart !== null && ($lastRealStart === null || $lastAnyStart > $lastRealStart)) { + $logger->debug('check-limits: silence skipped (maintenance sentinel was most recent event)', ['job_id' => $jobId]); + continue; + } + + // Calculate grace period: use per-job override or global default; also account for execution limit + $graceMinutes = $srow['silence_grace_minutes'] !== null + ? max(1, (int) $srow['silence_grace_minutes']) + : $globalGraceMinutes; + if ($srow['execution_limit_seconds'] !== null) { + $limitMinutes = (int) ceil((int) $srow['execution_limit_seconds'] / 60); + $graceMinutes = max($graceMinutes, $limitMinutes); + } + + // Calculate the expected previous run time via cron expression + try { + $cron = new CronExpression($schedule); + $expectedDt = $cron->getPreviousRunDate('now', 0, false, $tzName); + $expectedDtIm = \DateTimeImmutable::createFromMutable($expectedDt)->setTimezone($tz); + $silenceThreshold = $expectedDtIm->modify(sprintf('+%d minutes', $graceMinutes)); + } catch (\Throwable $e) { + $logger->warning('check-limits: could not parse schedule for silence detection', [ + 'job_id' => $jobId, + 'schedule' => $schedule, + 'message' => $e->getMessage(), + ]); + continue; + } + + $now = new \DateTimeImmutable('now', $tz); + + // Not yet past the silence threshold → job is not overdue + if ($now <= $silenceThreshold) { + continue; + } + + // Job ran at or after the expected time → not silent + if ($lastRealStart !== null && $lastRealStart >= $expectedDtIm->format('Y-m-d H:i:s')) { + continue; + } + + // Dedup: do not fire if an alert was sent in the last hour + if ($lastAlertAt !== null) { + try { + $lastAlertDt = new \DateTimeImmutable($lastAlertAt, $tz); + if ($lastAlertDt >= $now->modify('-1 hour')) { + $logger->debug('check-limits: silence alert suppressed (sent within last hour)', ['job_id' => $jobId]); + continue; + } + } catch (\Throwable) { + // Unparseable datetime → treat as no dedup, allow alert + } + } + + // Calculate how long the job has been silent + $referenceTime = $lastRealStart ?? $createdAt; + $silenceSeconds = max(0, time() - (int) strtotime($referenceTime)); + $silenceSinceMinutes = (int) round($silenceSeconds / 60); + $expectedLastRunStr = $expectedDtIm->format('Y-m-d H:i:s'); + $notifyTarget = count($targets) === 1 ? $targets[0] : implode(', ', $targets); + + // Update dedup timestamp before dispatching to prevent double-fire on slow SMTP + try { + $pdo->prepare( + 'UPDATE cronjobs SET last_silence_alert_at = NOW() WHERE id = :id' + )->execute([':id' => $jobId]); + } catch (\Throwable $e) { + $logger->error('check-limits: failed to update last_silence_alert_at', [ + 'job_id' => $jobId, + 'message' => $e->getMessage(), + ]); + } + + $logger->info('check-limits: dispatching silence alert', [ + 'job_id' => $jobId, + 'last_real_start' => $lastRealStart, + 'expected_last_run' => $expectedLastRunStr, + 'silence_since_minutes' => $silenceSinceMinutes, + ]); + + // Dispatch via background process (same pattern as limit-exceeded notification) + $payload = json_encode([ + 'type' => 'silence', + 'job_id' => $jobId, + 'description' => $description, + 'schedule' => $schedule, + 'last_started_at' => $lastRealStart, + 'expected_last_run' => $expectedLastRunStr, + 'silence_since_minutes' => $silenceSinceMinutes, + 'target' => $notifyTarget, + ], JSON_UNESCAPED_UNICODE); + + $dispatched = false; + + if ($payload !== false && file_exists($notifyScript) && $execAvailable) { + $tempFile = tempnam(sys_get_temp_dir(), 'cronmgr_silence_'); + if ($tempFile !== false && file_put_contents($tempFile, $payload) !== false) { + $cmd = sprintf( + 'timeout 30 php %s %s > /dev/null 2>&1 &', + escapeshellarg($notifyScript), + escapeshellarg($tempFile), + ); + exec($cmd); + $dispatched = true; + } + } + + if (!$dispatched) { + // Synchronous fallback + try { + $mailNotifier->sendSilenceAlert( + jobId: $jobId, + description: $description, + schedule: $schedule, + lastStartedAt: $lastRealStart, + expectedLastRun: $expectedLastRunStr, + silenceSinceMinutes: $silenceSinceMinutes, + target: $notifyTarget, + ); + } catch (\Throwable $e) { + $logger->error('check-limits: synchronous silence mail failed', [ + 'job_id' => $jobId, + 'message' => $e->getMessage(), + ]); + } + + try { + $telegramNotifier->sendSilenceAlert( + jobId: $jobId, + description: $description, + schedule: $schedule, + lastStartedAt: $lastRealStart, + expectedLastRun: $expectedLastRunStr, + silenceSinceMinutes: $silenceSinceMinutes, + target: $notifyTarget, + ); + } catch (\Throwable $e) { + $logger->error('check-limits: synchronous silence telegram failed', [ + 'job_id' => $jobId, + 'message' => $e->getMessage(), + ]); + } + } + } +} + $logger->debug('check-limits: finished'); exit(0); diff --git a/agent/bin/send-notification.php b/agent/bin/send-notification.php index 5ae2237..d587fb8 100755 --- a/agent/bin/send-notification.php +++ b/agent/bin/send-notification.php @@ -19,9 +19,17 @@ * immediately after reading to avoid leaving sensitive data * on disk. * - * Expected JSON payload keys: + * Expected JSON payload keys (type = "failure"): * job_id, description, linux_user, schedule, exit_code, output, - * started_at, finished_at + * started_at, finished_at, type, target, notify_after_failures, still_running + * + * Expected JSON payload keys (type = "recovery"): + * job_id, description, linux_user, schedule, started_at, finished_at, + * type, target, consecutive_failures + * + * Expected JSON payload keys (type = "silence"): + * job_id, description, schedule, last_started_at (nullable), expected_last_run, + * silence_since_minutes, type, target * * Exit codes: * 0 – mail dispatched successfully (or disabled in config) @@ -105,7 +113,33 @@ $mailNotifier = new MailNotifier($logger, $config); $telegramNotifier = new TelegramNotifier($logger, $config); - if ($type === 'recovery') { + if ($type === 'silence') { + $lastStartedAt = isset($data['last_started_at']) && $data['last_started_at'] !== null + ? (string) $data['last_started_at'] + : null; + $expectedLastRun = (string) ($data['expected_last_run'] ?? ''); + $silenceSinceMinutes = (int) ($data['silence_since_minutes'] ?? 0); + + $mailNotifier->sendSilenceAlert( + jobId: $jobId, + description: $description, + schedule: $schedule, + lastStartedAt: $lastStartedAt, + expectedLastRun: $expectedLastRun, + silenceSinceMinutes: $silenceSinceMinutes, + target: $target, + ); + + $telegramNotifier->sendSilenceAlert( + jobId: $jobId, + description: $description, + schedule: $schedule, + lastStartedAt: $lastStartedAt, + expectedLastRun: $expectedLastRun, + silenceSinceMinutes: $silenceSinceMinutes, + target: $target, + ); + } elseif ($type === 'recovery') { $consecutiveFailures = (int) ($data['consecutive_failures'] ?? 0); $mailNotifier->sendRecoveryAlert( @@ -129,7 +163,7 @@ finishedAt: $finishedAt, target: $target, ); - } else { + } else { // failure (default) $output = (string) ($data['output'] ?? ''); $notifyAfterFailures = max(1, (int) ($data['notify_after_failures'] ?? 1)); $stillRunning = (bool) ($data['still_running'] ?? false); diff --git a/agent/sql/migrations/016_silence_detection.sql b/agent/sql/migrations/016_silence_detection.sql new file mode 100644 index 0000000..474ae44 --- /dev/null +++ b/agent/sql/migrations/016_silence_detection.sql @@ -0,0 +1,14 @@ +-- Migration 016: Silence Detection +-- Adds three columns to cronjobs to support per-job "silence" alerting: +-- notify_on_silence – opt-in flag; when 1 an alert is sent when the job has +-- not started within its expected schedule window. +-- silence_grace_minutes – per-job override for the global silence.grace_minutes +-- config value; NULL means "use the global default". +-- last_silence_alert_at – timestamp of the most recent silence alert; reset to +-- NULL by ExecutionStartEndpoint when the job starts again. +-- Used to de-duplicate repeated alerts (max once per hour). + +ALTER TABLE cronjobs + ADD COLUMN notify_on_silence TINYINT(1) NOT NULL DEFAULT 0 AFTER notify_on_recovery, + ADD COLUMN silence_grace_minutes INT UNSIGNED NULL DEFAULT NULL AFTER notify_on_silence, + ADD COLUMN last_silence_alert_at DATETIME NULL DEFAULT NULL AFTER silence_grace_minutes; diff --git a/agent/src/Endpoints/CronCreateEndpoint.php b/agent/src/Endpoints/CronCreateEndpoint.php index e773474..2b834d6 100644 --- a/agent/src/Endpoints/CronCreateEndpoint.php +++ b/agent/src/Endpoints/CronCreateEndpoint.php @@ -157,6 +157,12 @@ public function handle(array $params): void $notifyAfterLimitExceeded = isset($body['notify_after_limit_exceeded']) && is_int($body['notify_after_limit_exceeded']) && $body['notify_after_limit_exceeded'] >= 1 ? $body['notify_after_limit_exceeded'] : 1; + $notifyOnSilence = isset($body['notify_on_silence']) && is_bool($body['notify_on_silence']) + ? $body['notify_on_silence'] + : false; + $silenceGraceMinutes = isset($body['silence_grace_minutes']) && is_int($body['silence_grace_minutes']) && $body['silence_grace_minutes'] >= 0 + ? $body['silence_grace_minutes'] + : null; $targets = $this->normaliseTargets($body['targets'] ?? ['local']); $tags = isset($body['tags']) && is_array($body['tags']) ? $body['tags'] : []; @@ -183,12 +189,16 @@ public function handle(array $params): void (linux_user, schedule, command, description, active, notify_on_failure, execution_limit_seconds, auto_kill_on_limit, singleton, run_in_maintenance, retention_days, retry_count, retry_delay_minutes, restart_on_exitcodes, - notify_after_failures, notify_after_limit_exceeded, execution_mode, ssh_host) + notify_after_failures, notify_after_limit_exceeded, + notify_on_silence, silence_grace_minutes, + execution_mode, ssh_host) VALUES (:linux_user, :schedule, :command, :description, :active, :notify_on_failure, :execution_limit_seconds, :auto_kill_on_limit, :singleton, :run_in_maintenance, :retention_days, :retry_count, :retry_delay_minutes, :restart_on_exitcodes, - :notify_after_failures, :notify_after_limit_exceeded, :execution_mode, :ssh_host)' + :notify_after_failures, :notify_after_limit_exceeded, + :notify_on_silence, :silence_grace_minutes, + :execution_mode, :ssh_host)' ); $stmt->execute([ ':linux_user' => $linuxUser, @@ -207,6 +217,8 @@ public function handle(array $params): void ':restart_on_exitcodes' => $restartOnExitcodes, ':notify_after_failures' => $notifyAfterFailures, ':notify_after_limit_exceeded' => $notifyAfterLimitExceeded, + ':notify_on_silence' => (int) $notifyOnSilence, + ':silence_grace_minutes' => $silenceGraceMinutes, ':execution_mode' => $executionMode, ':ssh_host' => $sshHost, ]); @@ -566,6 +578,9 @@ private function fetchJob(int $jobId): array j.retry_delay_minutes, j.notify_after_failures, j.notify_after_limit_exceeded, + j.notify_on_silence, + j.silence_grace_minutes, + j.last_silence_alert_at, j.execution_mode, j.ssh_host, j.created_at, @@ -614,6 +629,13 @@ private function fetchJob(int $jobId): array 'retry_delay_minutes' => (int) ($row['retry_delay_minutes'] ?? 1), 'notify_after_failures' => (int) ($row['notify_after_failures'] ?? 1), 'notify_after_limit_exceeded' => (int) ($row['notify_after_limit_exceeded'] ?? 1), + 'notify_on_silence' => (bool) ($row['notify_on_silence'] ?? false), + 'silence_grace_minutes' => isset($row['silence_grace_minutes']) && $row['silence_grace_minutes'] !== null + ? (int) $row['silence_grace_minutes'] + : null, + 'last_silence_alert_at' => isset($row['last_silence_alert_at']) && $row['last_silence_alert_at'] !== null + ? (string) $row['last_silence_alert_at'] + : null, 'targets' => $targets, // Legacy fields kept for backward compatibility 'execution_mode' => (string) ($row['execution_mode'] ?? 'local'), diff --git a/agent/src/Endpoints/CronGetEndpoint.php b/agent/src/Endpoints/CronGetEndpoint.php index 1bdb414..7c48a6b 100644 --- a/agent/src/Endpoints/CronGetEndpoint.php +++ b/agent/src/Endpoints/CronGetEndpoint.php @@ -186,6 +186,9 @@ private function fetchJob(int $jobId): ?array j.restart_on_exitcodes, j.notify_after_failures, j.notify_after_limit_exceeded, + j.notify_on_silence, + j.silence_grace_minutes, + j.last_silence_alert_at, j.execution_mode, j.ssh_host, j.created_at, @@ -272,6 +275,13 @@ private function normaliseRow(array $row): array : null, 'notify_after_failures' => max(1, (int) ($row['notify_after_failures'] ?? 1)), 'notify_after_limit_exceeded' => max(1, (int) ($row['notify_after_limit_exceeded'] ?? 1)), + 'notify_on_silence' => (bool) ($row['notify_on_silence'] ?? false), + 'silence_grace_minutes' => isset($row['silence_grace_minutes']) && $row['silence_grace_minutes'] !== null + ? (int) $row['silence_grace_minutes'] + : null, + 'last_silence_alert_at' => isset($row['last_silence_alert_at']) && $row['last_silence_alert_at'] !== null + ? (string) $row['last_silence_alert_at'] + : null, 'targets' => $targets, // Legacy fields kept so old wrapper invocations (no target arg) still work 'execution_mode' => (string) ($row['execution_mode'] ?? 'local'), diff --git a/agent/src/Endpoints/CronListEndpoint.php b/agent/src/Endpoints/CronListEndpoint.php index f486bee..b35703c 100644 --- a/agent/src/Endpoints/CronListEndpoint.php +++ b/agent/src/Endpoints/CronListEndpoint.php @@ -249,6 +249,9 @@ private function fetchJobs(?string $userFilter, ?string $tagFilter, ?string $tar j.restart_on_exitcodes, j.notify_after_failures, j.notify_after_limit_exceeded, + j.notify_on_silence, + j.silence_grace_minutes, + j.last_silence_alert_at, j.execution_mode, j.ssh_host, j.created_at, @@ -363,6 +366,13 @@ private function normaliseRow(array $row): array : null, 'notify_after_failures' => (int) ($row['notify_after_failures'] ?? 1), 'notify_after_limit_exceeded' => (int) ($row['notify_after_limit_exceeded'] ?? 1), + 'notify_on_silence' => (bool) ($row['notify_on_silence'] ?? false), + 'silence_grace_minutes' => isset($row['silence_grace_minutes']) && $row['silence_grace_minutes'] !== null + ? (int) $row['silence_grace_minutes'] + : null, + 'last_silence_alert_at' => isset($row['last_silence_alert_at']) && $row['last_silence_alert_at'] !== null + ? (string) $row['last_silence_alert_at'] + : null, 'last_run' => isset($row['last_run']) && $row['last_run'] !== null ? (string) $row['last_run'] : null, 'last_exit_code' => isset($row['last_exit_code']) && $row['last_exit_code'] !== null ? (int) $row['last_exit_code'] : null, ]; diff --git a/agent/src/Endpoints/CronUpdateEndpoint.php b/agent/src/Endpoints/CronUpdateEndpoint.php index 98bde9c..9b527dc 100644 --- a/agent/src/Endpoints/CronUpdateEndpoint.php +++ b/agent/src/Endpoints/CronUpdateEndpoint.php @@ -189,6 +189,8 @@ public function handle(array $params): void restart_on_exitcodes = :restart_on_exitcodes, notify_after_failures = :notify_after_failures, notify_after_limit_exceeded = :notify_after_limit_exceeded, + notify_on_silence = :notify_on_silence, + silence_grace_minutes = :silence_grace_minutes, execution_mode = :execution_mode, ssh_host = :ssh_host WHERE id = :id' @@ -213,6 +215,10 @@ public function handle(array $params): void : null, ':notify_after_failures' => max(1, (int) ($merged['notify_after_failures'] ?? 1)), ':notify_after_limit_exceeded' => max(1, (int) ($merged['notify_after_limit_exceeded'] ?? 1)), + ':notify_on_silence' => (int) ($merged['notify_on_silence'] ?? false), + ':silence_grace_minutes' => isset($merged['silence_grace_minutes']) && $merged['silence_grace_minutes'] !== null + ? max(0, (int) $merged['silence_grace_minutes']) + : null, ':execution_mode' => $executionMode, ':ssh_host' => $sshHost, ':id' => $jobId, @@ -372,6 +378,16 @@ private function mergeFields(array $existing, array $body): array 'notify_after_limit_exceeded' => array_key_exists('notify_after_limit_exceeded', $body) ? max(1, (int) $body['notify_after_limit_exceeded']) : max(1, (int) ($existing['notify_after_limit_exceeded'] ?? 1)), + 'notify_on_silence' => array_key_exists('notify_on_silence', $body) + ? (bool) $body['notify_on_silence'] + : (bool) ($existing['notify_on_silence'] ?? false), + 'silence_grace_minutes' => array_key_exists('silence_grace_minutes', $body) + ? (is_int($body['silence_grace_minutes']) && $body['silence_grace_minutes'] >= 0 + ? $body['silence_grace_minutes'] + : null) + : (isset($existing['silence_grace_minutes']) && $existing['silence_grace_minutes'] !== null + ? (int) $existing['silence_grace_minutes'] + : null), 'targets' => $mergedTargets, 'tags' => array_key_exists('tags', $body) ? $body['tags'] : $existingTags, ]; @@ -663,6 +679,9 @@ private function fetchJobRaw(int $jobId): ?array j.restart_on_exitcodes, j.notify_after_failures, j.notify_after_limit_exceeded, + j.notify_on_silence, + j.silence_grace_minutes, + j.last_silence_alert_at, j.execution_mode, j.ssh_host, j.created_at, @@ -729,6 +748,13 @@ private function fetchJob(int $jobId): array : null, 'notify_after_failures' => max(1, (int) ($row['notify_after_failures'] ?? 1)), 'notify_after_limit_exceeded' => max(1, (int) ($row['notify_after_limit_exceeded'] ?? 1)), + 'notify_on_silence' => (bool) ($row['notify_on_silence'] ?? false), + 'silence_grace_minutes' => isset($row['silence_grace_minutes']) && $row['silence_grace_minutes'] !== null + ? (int) $row['silence_grace_minutes'] + : null, + 'last_silence_alert_at' => isset($row['last_silence_alert_at']) && $row['last_silence_alert_at'] !== null + ? (string) $row['last_silence_alert_at'] + : null, 'targets' => $targets, 'execution_mode' => (string) ($row['execution_mode'] ?? 'local'), 'ssh_host' => isset($row['ssh_host']) ? (string) $row['ssh_host'] : null, diff --git a/agent/src/Endpoints/ExecutionStartEndpoint.php b/agent/src/Endpoints/ExecutionStartEndpoint.php index 4ff21fe..de75bd4 100644 --- a/agent/src/Endpoints/ExecutionStartEndpoint.php +++ b/agent/src/Endpoints/ExecutionStartEndpoint.php @@ -345,6 +345,13 @@ public function handle(array $params): void $executionId = (int) $this->pdo->lastInsertId(); + // Reset silence alert dedup flag so the next silence-detection cycle + // does not re-fire immediately after the job recovers. + $this->pdo->prepare( + 'UPDATE cronjobs SET last_silence_alert_at = NULL + WHERE id = :id AND last_silence_alert_at IS NOT NULL' + )->execute([':id' => $jobId]); + $this->logger->info('ExecutionStartEndpoint: execution started', [ 'execution_id' => $executionId, 'job_id' => $jobId, diff --git a/agent/src/Notification/MailNotifier.php b/agent/src/Notification/MailNotifier.php index 36a0452..c85594c 100644 --- a/agent/src/Notification/MailNotifier.php +++ b/agent/src/Notification/MailNotifier.php @@ -309,6 +309,76 @@ public function sendRecoveryAlert( } } + public function sendSilenceAlert( + int $jobId, + string $description, + string $schedule, + ?string $lastStartedAt, + string $expectedLastRun, + int $silenceSinceMinutes, + string $target = '', + ): bool { + $enabled = (bool) $this->config->get('mail.enabled', false); + + if (!$enabled) { + $this->logger->debug('MailNotifier: mail disabled in config, skipping silence alert', [ + 'job_id' => $jobId, + ]); + return false; + } + + $host = (string) $this->config->get('mail.host', 'smtp.example.com'); + $port = (int) $this->config->get('mail.port', 587); + $user = (string) $this->config->get('mail.username', ''); + $pass = (string) $this->config->get('mail.password', ''); + $fromAddr = (string) $this->config->get('mail.from', 'cronmanager@example.com'); + $fromName = (string) $this->config->get('mail.from_name', 'Cronmanager'); + $toAddr = (string) $this->config->get('mail.to', ''); + $encryption = (string) $this->config->get('mail.encryption', 'tls'); + $baseUrl = rtrim((string) $this->config->get('notifications.web_url', ''), '/'); + + try { + $mail = new \PHPMailer\PHPMailer\PHPMailer(true); + $mail->isSMTP(); + $mail->Host = $host; + $mail->SMTPAuth = ($user !== ''); + $mail->Username = $user; + $mail->Password = $pass; + $mail->SMTPSecure = $encryption === 'ssl' + ? \PHPMailer\PHPMailer\PHPMailer::ENCRYPTION_SMTPS + : \PHPMailer\PHPMailer\PHPMailer::ENCRYPTION_STARTTLS; + $mail->Port = $port; + $mail->CharSet = 'UTF-8'; + $mail->Timeout = (int) $this->config->get('mail.smtp_timeout', 15); + + $mail->setFrom($fromAddr, $fromName); + $mail->addAddress($toAddr); + $mail->Subject = sprintf('[Cronmanager] Job #%d SILENT: %s', $jobId, $description); + + $e = fn(string $s): string => htmlspecialchars($s, ENT_QUOTES, 'UTF-8'); + + $mail->Body = $this->buildSilenceHtmlBody($jobId, $description, $schedule, + $lastStartedAt, $expectedLastRun, $silenceSinceMinutes, $target, $baseUrl, $e); + $mail->AltBody = $this->buildSilencePlainBody($jobId, $description, $schedule, + $lastStartedAt, $expectedLastRun, $silenceSinceMinutes, $target, $baseUrl); + $mail->isHTML(true); + $mail->send(); + + $this->logger->info('MailNotifier: silence alert sent', [ + 'job_id' => $jobId, + 'to' => $toAddr, + ]); + + return true; + } catch (\Throwable $ex) { + $this->logger->error('MailNotifier: failed to send silence alert', [ + 'job_id' => $jobId, + 'message' => $ex->getMessage(), + ]); + return false; + } + } + public function sendTest(): array { $host = (string) $this->config->get('mail.host', 'smtp.example.com'); @@ -687,6 +757,123 @@ private function buildRecoveryPlainBody( return implode("\n", $lines); } + private function buildSilencePlainBody( + int $jobId, + string $description, + string $schedule, + ?string $lastStartedAt, + string $expectedLastRun, + int $silenceSinceMinutes, + string $target, + string $baseUrl, + ): string { + $lastStartLine = $lastStartedAt !== null + ? sprintf('Last seen : %s', $lastStartedAt) + : 'Last seen : never (job has not run yet)'; + + $hours = intdiv($silenceSinceMinutes, 60); + $minutes = $silenceSinceMinutes % 60; + $silent = $hours > 0 + ? sprintf('%dh %02dm', $hours, $minutes) + : sprintf('%dm', $minutes); + + $lines = [ + 'CRONMANAGER – JOB SILENCE ALERT', + str_repeat('=', 60), + '', + sprintf('Job ID : %d', $jobId), + sprintf('Description : %s', $description), + sprintf('Schedule : %s', $schedule), + ]; + + if ($target !== '' && $target !== 'local') { + $lines[] = sprintf('Target : %s', $target); + } + + $lines[] = sprintf('Expected at : %s', $expectedLastRun); + $lines[] = $lastStartLine; + $lines[] = sprintf('Silent for : %s', $silent); + $lines[] = ''; + $lines[] = 'The job has not started as scheduled. Please investigate.'; + $lines[] = ''; + + if ($baseUrl !== '') { + $lines[] = sprintf('View in Cronmanager: %s/crons/%d', $baseUrl, $jobId); + } + + return implode("\n", $lines); + } + + private function buildSilenceHtmlBody( + int $jobId, + string $description, + string $schedule, + ?string $lastStartedAt, + string $expectedLastRun, + int $silenceSinceMinutes, + string $target, + string $baseUrl, + callable $e, + ): string { + $targetRow = ($target !== '' && $target !== 'local') + ? "Target{$e($target)}" + : ''; + + $lastSeenCell = $lastStartedAt !== null + ? $e($lastStartedAt) + : 'never (job has not run yet)'; + + $hours = intdiv($silenceSinceMinutes, 60); + $minutes = $silenceSinceMinutes % 60; + $silent = $hours > 0 + ? sprintf('%dh %02dm', $hours, $minutes) + : sprintf('%dm', $minutes); + + $linkHtml = $baseUrl !== '' + ? sprintf( + '

🔗 View in Cronmanager

', + $e($baseUrl), + $jobId + ) + : ''; + + return << + + + + Cronmanager – Job Silence Alert + + + +
+

🤐 Job Silence Alert

+

The job has not started as scheduled. Please investigate.

+ + + + + {$targetRow} + + + +
Job ID {$e((string)$jobId)}
Description {$e($description)}
Schedule {$e($schedule)}
Expected at {$e($expectedLastRun)}
Last seen {$lastSeenCell}
Silent for {$silent}
+ {$linkHtml} +

Cronmanager – automated job monitoring

+
+ + +HTML; + } + private function buildRecoveryHtmlBody( int $jobId, string $description, diff --git a/agent/src/Notification/TelegramNotifier.php b/agent/src/Notification/TelegramNotifier.php index 9bb5d8b..b4e62fb 100644 --- a/agent/src/Notification/TelegramNotifier.php +++ b/agent/src/Notification/TelegramNotifier.php @@ -309,6 +309,99 @@ public function sendRecoveryAlert( } } + public function sendSilenceAlert( + int $jobId, + string $description, + string $schedule, + ?string $lastStartedAt, + string $expectedLastRun, + int $silenceSinceMinutes, + string $target = '', + ): bool { + $enabled = (bool) $this->config->get('telegram.enabled', false); + + if (!$enabled) { + $this->logger->debug('TelegramNotifier: disabled in config, skipping silence alert', [ + 'job_id' => $jobId, + ]); + return false; + } + + $botToken = (string) $this->config->get('telegram.bot_token', ''); + $chatId = (string) $this->config->get('telegram.chat_id', ''); + $timeout = (int) $this->config->get('telegram.timeout', 15); + $baseUrl = rtrim((string) $this->config->get('notifications.web_url', ''), '/'); + + if ($botToken === '' || $chatId === '') { + $this->logger->warning('TelegramNotifier: bot_token or chat_id not configured, skipping silence alert', [ + 'job_id' => $jobId, + ]); + return false; + } + + $targetLine = ($target !== '' && $target !== 'local') + ? "\n🎯 Target: " . htmlspecialchars($target, ENT_QUOTES, 'UTF-8') + : ''; + + $hours = intdiv($silenceSinceMinutes, 60); + $minutes = $silenceSinceMinutes % 60; + $silent = $hours > 0 + ? sprintf('%dh %02dm', $hours, $minutes) + : sprintf('%dm', $minutes); + + $lastSeenLine = $lastStartedAt !== null + ? htmlspecialchars($lastStartedAt, ENT_QUOTES, 'UTF-8') + : 'never (job has not run yet)'; + + $linkLine = $baseUrl !== '' + ? sprintf("\n\n🔗 View in Cronmanager", htmlspecialchars($baseUrl, ENT_QUOTES, 'UTF-8'), $jobId) + : ''; + + $text = sprintf( + "🤐 Job Silence Alert\n\n" + . "🆔 Job #%d: %s\n" + . "⏰ Schedule: %s%s\n" + . "⏳ Expected at: %s\n" + . "👀 Last seen: %s\n" + . "🚫 Silent for: %s%s", + $jobId, + htmlspecialchars($description, ENT_QUOTES, 'UTF-8'), + htmlspecialchars($schedule, ENT_QUOTES, 'UTF-8'), + $targetLine, + htmlspecialchars($expectedLastRun, ENT_QUOTES, 'UTF-8'), + $lastSeenLine, + $silent, + $linkLine, + ); + + if (\mb_strlen($text) > self::MAX_MESSAGE_LENGTH) { + $text = \mb_substr($text, 0, self::MAX_MESSAGE_LENGTH - 3) . '...'; + } + + try { + $client = new Client(['timeout' => $timeout]); + $client->post(self::API_BASE . $botToken . '/sendMessage', [ + 'json' => [ + 'chat_id' => $chatId, + 'text' => $text, + 'parse_mode' => 'HTML', + ], + ]); + + $this->logger->info('TelegramNotifier: silence alert sent', [ + 'job_id' => $jobId, + ]); + + return true; + } catch (GuzzleException $e) { + $this->logger->error('TelegramNotifier: failed to send silence alert', [ + 'job_id' => $jobId, + 'message' => $e->getMessage(), + ]); + return false; + } + } + // ------------------------------------------------------------------------- // Test API // ------------------------------------------------------------------------- diff --git a/web/VERSION b/web/VERSION index cbe06cd..a84947d 100644 --- a/web/VERSION +++ b/web/VERSION @@ -1 +1 @@ -4.4.4 +4.5.0 diff --git a/web/lang/de.php b/web/lang/de.php index 7ea6b65..23e37f9 100644 --- a/web/lang/de.php +++ b/web/lang/de.php @@ -140,6 +140,12 @@ 'cron_notify_after_limit_exceeded' => 'Benachrichtigen nach N aufeinanderfolgenden Limit-Überschreitungen', 'cron_notify_after_limit_exceeded_unit' => 'aufeinanderfolgende Limit-Überschreitungen', 'cron_notify_after_limit_exceeded_hint' => 'Limit-Benachrichtigung erst nach dieser Anzahl aufeinanderfolgender Laufzeitüberschreitungen senden (Standard: 1 = jedes Mal). Nach der Benachrichtigung werden keine weiteren Alerts gesendet, bis eine Ausführung innerhalb des Limits abgeschlossen wird.', + 'cron_notify_on_silence' => 'Silence-Alarm aktivieren', + 'cron_notify_on_silence_hint' => 'Benachrichtigung senden, wenn der Job seinen geplanten Startzeitpunkt überschritten hat ohne gestartet zu sein. Schützt vor komplettem Ausfall ohne sichtbares Fehlerprotokoll.', + 'cron_silence_grace_minutes' => 'Toleranzzeit (Minuten)', + 'cron_silence_grace_unit' => 'Minuten', + 'cron_silence_grace_default' => 'Standard (global)', + 'cron_silence_grace_hint' => 'Puffer nach dem geplanten Start, bevor ein Alarm ausgelöst wird. Leer lassen für den globalen Standardwert (silence.grace_minutes in config.json).', 'cron_retry_badge' => 'Versuch {attempt}/{total}', 'cron_singleton' => 'Singleton', 'cron_singleton_hint' => 'Neue Ausführungen überspringen, solange eine vorherige Instanz noch läuft.', diff --git a/web/lang/en.php b/web/lang/en.php index 2281afb..8a35f59 100644 --- a/web/lang/en.php +++ b/web/lang/en.php @@ -140,6 +140,12 @@ 'cron_notify_after_limit_exceeded' => 'Notify after N consecutive limit exceedings', 'cron_notify_after_limit_exceeded_unit' => 'consecutive limit exceedings', 'cron_notify_after_limit_exceeded_hint' => 'Send a limit-exceeded alert only after this many consecutive executions have exceeded the runtime limit (default: 1 = every time). Once notified, no further alerts are sent until an execution completes within the limit.', + 'cron_notify_on_silence' => 'Enable silence detection', + 'cron_notify_on_silence_hint' => 'Send a notification when the job has missed its scheduled start time. Protects against a complete outage with no visible failure log entry.', + 'cron_silence_grace_minutes' => 'Grace period (minutes)', + 'cron_silence_grace_unit' => 'minutes', + 'cron_silence_grace_default' => 'default (global)', + 'cron_silence_grace_hint' => 'Buffer after the scheduled start before an alert fires. Leave empty to use the global default (silence.grace_minutes in config.json).', 'cron_retry_badge' => 'Retry {attempt}/{total}', 'cron_singleton' => 'Singleton', 'cron_singleton_hint' => 'Skip new executions while a previous instance is still running.', diff --git a/web/src/Controller/CronController.php b/web/src/Controller/CronController.php index 57f4698..da43186 100644 --- a/web/src/Controller/CronController.php +++ b/web/src/Controller/CronController.php @@ -1208,6 +1208,12 @@ private function buildJobPayload(array $post): array ? (int) $rawNotifyAfterLimit : 1; + // silence_grace_minutes: positive integer or null (use global default) + $rawSilenceGrace = trim((string) ($post['silence_grace_minutes'] ?? '')); + $silenceGraceMinutes = ($rawSilenceGrace !== '' && ctype_digit($rawSilenceGrace) && (int) $rawSilenceGrace > 0) + ? (int) $rawSilenceGrace + : null; + return [ 'linux_user' => trim((string) ($post['linux_user'] ?? '')), 'schedule' => trim((string) ($post['schedule'] ?? '')), @@ -1217,6 +1223,8 @@ private function buildJobPayload(array $post): array 'active' => isset($post['active']), 'notify_on_failure' => isset($post['notify_on_failure']), 'notify_on_recovery' => isset($post['notify_on_recovery']), + 'notify_on_silence' => isset($post['notify_on_silence']), + 'silence_grace_minutes' => $silenceGraceMinutes, 'execution_limit_seconds' => $executionLimitSeconds, 'auto_kill_on_limit' => isset($post['auto_kill_on_limit']), 'singleton' => isset($post['singleton']), diff --git a/web/templates/cron/form.php b/web/templates/cron/form.php index 6662838..9da48f4 100644 --- a/web/templates/cron/form.php +++ b/web/templates/cron/form.php @@ -49,6 +49,7 @@ $isActiveVal = $job !== null ? !empty($job['active']) : true; $isNotifyVal = $job !== null ? !empty($job['notify_on_failure']) : true; $isRecoveryVal = $job !== null ? !empty($job['notify_on_recovery']) : true; +$isSilenceVal = $job !== null ? !empty($job['notify_on_silence']) : false; $isAutoKillVal = $job !== null ? !empty($job['auto_kill_on_limit']) : false; $isSingletonVal = $job !== null ? !empty($job['singleton']) : false; $isRunInMaintVal = $job !== null ? !empty($job['run_in_maintenance']) : false; @@ -66,6 +67,7 @@ // Auto-expand the notify tab when any notify setting deviates from the default. $notifyOpen = !$isNotifyVal || !$isRecoveryVal + || $isSilenceVal || (int) $val('notify_after_failures', '1') > 1 || (int) $val('notify_after_limit_exceeded', '1') > 1; @@ -561,6 +563,47 @@ class="w-20 border border-gray-300 dark:border-gray-600 rounded-lg px-3 py-2 tex

+ +
+ +

+ +

+
+ + +
+ +
+ + + + +
+

+ +

+
+ @@ -907,4 +950,16 @@ function debouncedCheck() { } }); })(); + +// notify_on_silence → show/hide silence_grace_minutes row +(function () { + const silenceBox = document.getElementById('notify_on_silence'); + const graceRow = document.getElementById('silence-grace-row'); + + if (!silenceBox || !graceRow) { return; } + + silenceBox.addEventListener('change', function () { + graceRow.style.display = this.checked ? '' : 'none'; + }); +})();