[BUG] Notification channels configured, receiving notifications and test notification but not receiving digest even its also configured

[dbius]

Description

I configured Proxmenux Monitor to send notifications on Telegram and email. Both works, receiving notifications and test notifications, too.

Steps to Reproduce

Also configured to receive digest at 09:00 but I am not receiving it. Tried to restart only proxmenux-monitor, whole Proxmox PVE Host system, still not receiving daily digest notification.

Expected Behavior

To receive digest notification daily at 09:00 AM

Screenshots (Required)

Environment

• Operating system: Proxmox Virtual Environment 9.2.3
• Other relevant details: Proxmenux Monitor version 1.2.2

Additional Information

Add any other context about the problem here.

here are some other information that may help:

root@proxmox:~# curl -sk https://127.0.0.1:8008/api/notifications/status \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool
{
    "channels": {
        "email": {
            "connected": true,
            "type": "email"
        },
        "telegram": {
            "connected": true,
            "type": "telegram"
        }
    },
    "enabled": true,
    "running": true,
    "stats": {
        "last_sent_at": null,
        "started_at": "2026-06-15T21:55:34.913550",
        "total_errors": 0,
        "total_sent": 0
    },
    "watchers": {
        "journal": true,
        "polling": true,
        "task": true
    }
}
root@proxmox:~# curl -sk https://127.0.0.1:8008/api/notifications/history \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool | head -80
{
    "history": [
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "auth_fail",
            "id": 21,
            "message": "Failed login attempt detected.\n\ud83c\udf10 Source IP: 192.168.1.131\nUser: root\n\u2699\ufe0f Service: pam",
            "sent_at": "2026-06-15T20:25:39.963052",
            "severity": "WARNING",
            "source": "journal",
            "success": 1,
            "title": "\ud83d\udea8 ProxMenux Monitor: Authentication failure"
        },
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "error_resolved",
            "id": 20,
            "message": "The storage issue has been resolved.\nCT 107 (transmission) rootfs at 94.7% (7555 MB / 7981 MB) (recovered)\n\ud83d\udea6 Previous severity: WARNING\n\u23f1\ufe0f Duration: 5m",
            "sent_at": "2026-06-14T19:42:28.127485",
            "severity": "OK",
            "source": "health",
            "success": 1,
            "title": "\u2705 ProxMenux Monitor: Resolved - storage"
        },
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "health_degraded",
            "id": 19,
            "message": "1 CT(s) at >85.0% rootfs: CT 107 (transmission)",
            "sent_at": "2026-06-14T19:36:34.926153",
            "severity": "WARNING",
            "source": "health_monitor",
            "success": 1,
            "title": "\u26a0\ufe0f proxmox: Health check degraded"
        },
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "lxc_disk_low",
            "id": 18,
            "message": "CT 107 (transmission) rootfs is at 94.7% (7922962432 / 8369172480).\n\nA full LXC rootfs prevents the container from booting cleanly. Either expand the rootfs (pct resize 107 rootfs +1G) or free space inside the container.",
            "sent_at": "2026-06-14T19:36:34.168479",
            "severity": "WARNING",
            "source": "health",
            "success": 1,
            "title": "\ud83d\udcbe ProxMenux Monitor: CT 107 rootfs at 94.7%"
        },
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "ct_start",
            "id": 17,
            "message": "Container uptimekuma (ID: 111) is now running.",
            "sent_at": "2026-06-13T16:19:58.904333",
            "severity": "INFO",
            "source": "tasks",
            "success": 1,
            "title": "\u25b6\ufe0f ProxMenux Monitor: CT uptimekuma (111) started"
        },
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "auth_fail",
            "id": 16,
            "message": "Failed login attempt detected.\n\ud83c\udf10 Source IP: ::ffff:192.168.1.46\nUser: root\n\u2699\ufe0f Service: pam",
            "sent_at": "2026-06-13T14:15:10.395985",
            "severity": "WARNING",
            "source": "journal",
            "success": 1,
            "title": "\ud83d\udea8 ProxMenux Monitor: Authentication failure"
        },
        {
            "channel": "telegram",
            "error_message": null,
            "event_type": "ct_start",
            "id": 15,
            "message": "Container bentopdf (ID: 103) is now running.",
root@proxmox:~# curl -sk https://127.0.0.1:8008/api/ \
  -H "Authorization: Bearer $TOKEN" | python3 -m json.tool 2>/dev/null | head -30
root@proxmox:~#

root@proxmox:~# journalctl -u proxmenux-monitor -n 30 --no-pager
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/lib/python3.13/socketserver.py", line 845, in write
Jun 14 00:24:36 proxmox AppRun[1736773]:     self._sock.sendall(b)
Jun 14 00:24:36 proxmox AppRun[1736773]:     ~~~~~~~~~~~~~~~~~~^^^
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/lib/python3.13/ssl.py", line 1263, in sendall
Jun 14 00:24:36 proxmox AppRun[1736773]:     v = self.send(byte_view[count:])
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/lib/python3.13/ssl.py", line 1232, in send
Jun 14 00:24:36 proxmox AppRun[1736773]:     return self._sslobj.write(data)
Jun 14 00:24:36 proxmox AppRun[1736773]:            ~~~~~~~~~~~~~~~~~~^^^^^^
Jun 14 00:24:36 proxmox AppRun[1736773]: BrokenPipeError: [Errno 32] Broken pipe
Jun 14 00:24:36 proxmox AppRun[1736773]: During handling of the above exception, another exception occurred:
Jun 14 00:24:36 proxmox AppRun[1736773]: Traceback (most recent call last):
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/local/share/proxmenux/monitor-app/usr/lib/python3/dist-packages/werkzeug/serving.py", line 371, in run_wsgi
Jun 14 00:24:36 proxmox AppRun[1736773]:     execute(self.server.app)
Jun 14 00:24:36 proxmox AppRun[1736773]:     ~~~~~~~^^^^^^^^^^^^^^^^^
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/local/share/proxmenux/monitor-app/usr/lib/python3/dist-packages/werkzeug/serving.py", line 356, in execute
Jun 14 00:24:36 proxmox AppRun[1736773]:     data = self.rfile.read(10_000_000)
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/lib/python3.13/socket.py", line 719, in readinto
Jun 14 00:24:36 proxmox AppRun[1736773]:     return self._sock.recv_into(b)
Jun 14 00:24:36 proxmox AppRun[1736773]:            ~~~~~~~~~~~~~~~~~~~~^^^
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/lib/python3.13/ssl.py", line 1304, in recv_into
Jun 14 00:24:36 proxmox AppRun[1736773]:     return self.read(nbytes, buffer)
Jun 14 00:24:36 proxmox AppRun[1736773]:            ~~~~~~~~~^^^^^^^^^^^^^^^^
Jun 14 00:24:36 proxmox AppRun[1736773]:   File "/usr/lib/python3.13/ssl.py", line 1138, in read
Jun 14 00:24:36 proxmox AppRun[1736773]:     return self._sslobj.read(len, buffer)
Jun 14 00:24:36 proxmox AppRun[1736773]:            ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Jun 14 00:24:36 proxmox AppRun[1736773]: ssl.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] unexpected eof while reading (_ssl.c:2649)
Jun 15 20:57:57 proxmox systemd[1]: Stopping proxmenux-monitor.service - ProxMenux Monitor - Web Dashboard...
Jun 15 20:57:57 proxmox systemd[1]: proxmenux-monitor.service: Deactivated successfully.
Jun 15 20:57:57 proxmox systemd[1]: Stopped proxmenux-monitor.service - ProxMenux Monitor - Web Dashboard.
Jun 15 20:57:57 proxmox systemd[1]: proxmenux-monitor.service: Consumed 3h 53min 5.676s CPU time, 1.5G memory peak.
root@proxmox:~#

root@proxmox:~# systemctl status proxmenux-monitor
● proxmenux-monitor.service - ProxMenux Monitor - Web Dashboard
     Loaded: loaded (/etc/systemd/system/proxmenux-monitor.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-06-15 21:55:27 CEST; 2 days ago
 Invocation: db6a6b0b467f4e2bac4b030edd20c37b
   Main PID: 954 (python3)
      Tasks: 16 (limit: 34881)
     Memory: 183M (peak: 898.9M, swap: 42.2M, swap peak: 65.6M)
        CPU: 1h 43min 21.962s
     CGroup: /system.slice/proxmenux-monitor.service
             ├─    954 python3 /usr/local/share/proxmenux/monitor-app/usr/bin/flask_server.py
             ├─   1530 journalctl -f -o json --no-pager --cursor-file=/usr/local/share/proxmenux/journal_cursor.txt
             ├─2427141 /bin/bash -li
             ├─2427982 systemctl status proxmenux-monitor
             └─2427983 less

Jun 18 11:16:38 proxmox AppRun[954]:   File "/usr/lib/python3.13/socket.py", line 719, in readinto
Jun 18 11:16:38 proxmox AppRun[954]:     return self._sock.recv_into(b)
Jun 18 11:16:38 proxmox AppRun[954]:            ~~~~~~~~~~~~~~~~~~~~^^^
Jun 18 11:16:38 proxmox AppRun[954]:   File "/usr/lib/python3.13/ssl.py", line 1304, in recv_into
Jun 18 11:16:38 proxmox AppRun[954]:     return self.read(nbytes, buffer)
Jun 18 11:16:38 proxmox AppRun[954]:            ~~~~~~~~~^^^^^^^^^^^^^^^^
Jun 18 11:16:38 proxmox AppRun[954]:   File "/usr/lib/python3.13/ssl.py", line 1138, in read
Jun 18 11:16:38 proxmox AppRun[954]:     return self._sslobj.read(len, buffer)
Jun 18 11:16:38 proxmox AppRun[954]:            ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
Jun 18 11:16:38 proxmox AppRun[954]: ssl.SSLError: [SSL: UNEXPECTED_EOF_WHILE_READING] unexpected eof while reading (_ssl.c:2649)
root@proxmox:~#

[dbius]

After further investigation, individual notifications work correctly (ct_restart, ct_start, ct_stop, auth_fail, test — all appear in history). However, event_type: digest never appears in the notification history, even after multiple configured digest times have passed. The digest scheduler appears to not trigger at all. API stats now show total_sent: 11 but none of them are digest type.

root@proxmox:~# sqlite3 /usr/local/share/proxmenux/monitor.db ".tables"
disk_temperature_history  latency_history           temperature_history
root@proxmox:~# 

root@proxmox:~# curl -sk https://127.0.0.1:8008/api/notifications/status -H "Authorization: Bearer $TOKEN"
{"channels":{"email":{"connected":true,"type":"email"},"telegram":{"connected":true,"type":"telegram"}},"enabled":true,"running":true,"stats":{"last_sent_at":"2026-06-18T10:48:27.110555","started_at":"2026-06-15T21:55:34.913550","total_errors":0,"total_sent":11},"watchers":{"journal":true,"polling":true,"task":true}}
root@proxmox:~# 

[MacRimi]

Hi @dbius, thanks for the detailed report. I’ll review the code to see if I can spot where the message chain is breaking and why the notifications aren’t going through.

[MacRimi]

Hi @dbius, fixed in the upcoming build — thank you for the thorough report. The total_sent: 11 / total_errors: 0 output you pasted made the bug trivial to spot: the dispatcher was firing the digest and talking to the channel, but skipping the bookkeeping every other path uses.

Root cause in notification_manager.py::_flush_digest_for_channel:

• channel.send(...) was invoked but its result was never recorded in notification_history and never counted toward total_sent / total_errors — unlike _dispatch_to_channels, which calls _record_history(...) + bumps the counters on every send. That's why event_type: digest never showed up in your /api/notifications/history and total_sent stayed flat even though the schedule was firing.
• The buffer rows were deleted whether the send succeeded or returned {success: False} — a silent transport failure would still wipe the day's data.
• When the buffer was empty at the scheduled minute, the function returned with no trace at all, so the schedule looked dead even when it was actually working.

What changed:

• Every code path (success, fail, empty buffer) now calls _record_history('digest', ...) and updates total_sent / total_errors. The digest is finally visible from /api/notifications/history.
• Buffer rows are deleted only on a confirmed successful send.
• Empty-buffer fires now log a digest history entry that says "No INFO events buffered for this digest window" — so even on a quiet day you can confirm the scheduler ran on time.

A heads-up about what actually lands in your Telegram/email: the digest only buffers INFO events that aren't in _DIGEST_EXEMPT_EVENTS (vm_start, ct_start, system_reboot, etc. pass through immediately). Your history shows mostly WARNING (auth_fail, lxc_disk_low) and INFO events that are exempt (ct_start), so on a typical day the digest may legitimately have no rows to summarize — that case is now visible in the history feed instead of silent.

Available in the next beta, no setting needed.

Thanks again for the detailed diagnosis. 🙏

[dbius]

Hi @MacRimi,

Thank you so much for your answer and for all the hard work you did with proxmenux!

It's simply an amazing tool and should be essential for every Proxmox user!

Best wishes!
dbius