Monitoring Dashboard

The Monitoring page shows a live feed of every incident detected and investigated by the agent — whether triggered by the proactive poller, an EventBridge event, or a manual test. This page explains both the UI and the two detection systems that feed it.

Detection Modes

OpenDevOps Agent has two complementary detection modes. Both write to the same /monitoring page and can deliver to Slack and Telegram.

	Proactive Polling	Event-Driven (EventBridge → SQS)
How it works	Agent polls CloudWatch alarms + Lambda error rates on a timer	AWS fires EventBridge rules → SQS → consumer → agent
Setup	Set `POLL_INTERVAL_SECONDS` in `.env`	Create infrastructure via Settings → AWS Configuration
Detection latency	Up to N minutes (your poll interval)	2–5 min for alarm-based events; near-instant for direct events (ECS, RDS, GuardDuty)
CloudWatch alarm required	No — reads Lambda error rate directly via API	Only for metric/alarm detections; direct AWS service events do not need a CloudWatch alarm
Works without AWS setup	Yes — no SQS/EventBridge needed	No — requires SQS queue + EventBridge rules
Event types covered	CloudWatch alarms in ALARM state + Lambda error rates	CloudWatch alarms, ECS task failures, Lambda async errors, RDS events, EC2 state changes, CodePipeline failures, GuardDuty findings, AWS Health events
Appears in /monitoring	Yes	Yes
Slack notifications	Yes (if `SLACK_WEBHOOK_URL` set)	Yes (if `SLACK_WEBHOOK_URL` set)
Telegram notifications	Yes (if `TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID` set)	Yes (if `TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID` set)

Recommendation: enable both. Polling works immediately with zero AWS infrastructure and is faster for Lambda error detection. Event-driven catches a wider surface area (ECS, RDS, EC2, GuardDuty, etc.) once infrastructure is created.

Proactive Polling

What it does

Every POLL_INTERVAL_SECONDS seconds, the background poller runs two checks:

CloudWatch alarms — fetches every alarm currently in ALARM state; runs a full agent investigation for each new one
Lambda error rates — checks the last-hour error rate for up to 20 Lambda functions; runs an investigation for any function above POLL_ERROR_THRESHOLD%

Flow

Background timer fires every N seconds
  → get_alarms("ALARM")        — any alarm in ALARM state?
  → list_lambda_functions()    — check each function's error rate
      → error_rate > threshold?
          → agent.ainvoke()    — full ReAct investigation
              → add_alert()    — persists to alerts table → /monitoring
              → notify_slack() — posts to Slack

When to use

You want incident detection with zero AWS infrastructure setup
Fast Lambda error detection (1-minute polling catches errors in ~1–2 minutes)
You don't have CloudWatch alarms pre-configured

Configuration

POLL_INTERVAL_SECONDS=300     # 0 = disabled (default)
POLL_ERROR_THRESHOLD=5.0      # Lambda error rate % to trigger investigation
POLL_REINVESTIGATE_HOURS=1    # cooldown — skip re-investigating the same alarm within N hours

Set POLL_INTERVAL_SECONDS=0 to disable entirely. The poller only starts when this value is greater than zero.

Deduplication

The poller builds canonical incident keys and atomically claims them in the database before running the agent. POLL_REINVESTIGATE_HOURS controls how long a completed claim blocks another investigation for the same alarm or Lambda metric incident.

Because claims are durable, autonomous monitoring requires CHECKPOINT_BACKEND=sqlite or postgres. Memory mode is still useful for chat and quick demos, but the poller and event consumer are disabled there.

Source

src/agent/poller.py — see also slack_and_polling.md for Slack integration details.

Event-Driven Detection

What it does

An SQS long-poll consumer receives real AWS infrastructure events forwarded by EventBridge rules, runs a deterministic context-collection pass, then runs a full agent investigation.

Flow

AWS service emits an event (ECS task stopped, Lambda errors, RDS failover…)
  → CloudWatch / EventBridge evaluates matching rule
  → SQS queue (opendevops-agent-events)
      → event_consumer_loop() — long-polls every 20s
          → _is_real_failure() — filters noise (e.g. EC2 "running", healthy RDS events)
          → collect_context()  — deterministic boto3 pre-fetch (logs, metrics, config)
          → agent.ainvoke()    — full ReAct investigation
              → add_alert()    — persists to alerts table → /monitoring
              → notify_slack() — posts to Slack (+ Telegram if configured)

When to use

You want broad coverage beyond Lambda (ECS, RDS, EC2, GuardDuty, AWS Health)
You already have CloudWatch alarms configured on your resources
You want near-real-time detection instead of polling-interval latency

Why 2–5 minutes for alarm-based events?

CloudWatch alarm-triggered EventBridge events have an inherent delay:

Lambda generates errors → CloudWatch collects metric (~1 min)
Alarm evaluation period fires (minimum 60s period)
Alarm transitions OK → ALARM → EventBridge fires (near-instant)
SQS consumer picks up message (up to 20s long-poll)
Agent investigation runs (30–90s)

Direct events (ECS task stopped, GuardDuty finding, RDS failover) skip the alarm evaluation and arrive in seconds.

Event types and rules

EventBridge Rule	Events captured
`opendevops-alarm-state`	CloudWatch Alarm state changes (OK → ALARM)
`opendevops-lambda-failure`	Lambda async invocation failures
`opendevops-lambda-throttle`	Lambda throttling errors
`opendevops-ecs-task-stopped`	ECS task stopped with non-zero exit / OOM
`opendevops-ec2-state`	EC2 instance terminated
`opendevops-rds-events`	RDS failures, failovers, recovery
`opendevops-health`	AWS Health personal health events
`opendevops-codedeploy-failure`	CodeDeploy deployment failures
`opendevops-guardduty`	GuardDuty findings

Aggregate Lambda alarm

When you create infrastructure (Settings → AWS Configuration → Create Infrastructure), the setup wizard also creates a CloudWatch alarm named opendevops-lambda-errors-aggregate. This alarm:

Monitors AWS/Lambda Errors metric with no dimensions — meaning it watches ALL Lambda functions in the account
Trips when any Lambda has ≥ 1 error in a 60-second window
Triggers the opendevops-alarm-state EventBridge rule → SQS → consumer

This means you get event-driven Lambda coverage without creating per-function alarms. Because it is account-wide and trips on a single error, it is intentionally easy to test but can be noisy in busy accounts. For production, consider replacing it with scoped per-function or per-service alarms once you know which workloads matter.

Setup

Go to Settings → AWS Configuration → Create Infrastructure. This creates:

SQS queue: opendevops-agent-events
SQS DLQ: opendevops-agent-events-dlq
9 EventBridge rules (all prefixed opendevops-)
Aggregate CloudWatch alarm: opendevops-lambda-errors-aggregate

Wizard and infrastructure state is stored in the database for SQLite/PostgreSQL deployments (app_config). Memory mode does not support autonomous monitoring. Teardown deletes only infrastructure created by the wizard; queues supplied through .env or pasted manually are disconnected from the app but not deleted.

To remove it, click Teardown. See event_detection.md for full setup details and IAM permission requirements.

Context collectors

Before the LLM runs, collect_context() makes deterministic boto3 calls to pre-fetch facts about the affected resource (logs, metrics, config). This reduces LLM tool calls by front-loading the most relevant data. Capped at 3,000 chars to stay within token budgets.

Source

src/agent/event_consumer.py, src/agent/event_infra.py, src/agent/context_collectors.py — see also event_detection.md.

Choosing Between the Two

Scenario	Recommendation
Getting started, no AWS setup	Polling only (`POLL_INTERVAL_SECONDS=300`)
Lambda error monitoring only	Polling — faster and simpler
ECS, RDS, EC2, GuardDuty coverage	Event-driven required
Already have CloudWatch alarms	Event-driven — reuses your existing alarms
Want fastest Lambda detection	Both — polling at 1 min catches errors before the alarm trips

Testing the Pipeline

Test polling — set POLL_INTERVAL_SECONDS=60, trigger Lambda errors, wait ~1–2 minutes.

Test event-driven (fast — bypasses CloudWatch):

uv run python scripts/test_pipeline.py --function my-function

Pushes a synthetic alarm event directly to SQS. Result appears on /monitoring in ~60–90s.

Test event-driven (real path — waits for alarm):

uv run python scripts/test_pipeline.py --function my-function --lambda-only

Invokes the Lambda with a bad payload, then exits. The opendevops-lambda-errors-aggregate alarm trips in ~1–2 minutes and fires the full EventBridge → SQS → agent path.

Monitoring Dashboard UI

Incident Feed

Each alert card shows:

Status badge — HIGH (red), MEDIUM (amber), LOW (grey), or FAILED (rose) if the agent hit its tool limit or encountered an error
Service — the affected AWS service and resource name
Error — the root cause summary from the agent
Time — when the event was detected

Alerts are sorted by most recent first. Click any card to open the Alert Detail page.

Alert Detail Page

Shows the full investigation result:

Root cause in a highlighted panel
Resolution steps produced by the agent
Timestamp and detection source

Investigate / View investigation button — behaviour depends on whether the investigation session was persisted:

View investigation — shown for event-driven and poller alerts. Opens the original investigation chat so you can see the agent's tool calls, reasoning, and evidence, and send follow-up messages.
Investigate — shown for older alerts (created before session persistence was added). Opens a new chat with a pre-seeded prompt for deeper analysis.

Session continuity and sidebar

When the event consumer or poller runs an investigation, it creates a session (source = 'event') and links it to the resulting alert via session_id. These sessions are hidden from the sidebar by default — they represent automated runs the user did not initiate.

If you open a "View investigation" chat and send at least one follow-up message, the session is promoted to the sidebar (user_interacted = true) so you can navigate back to it without going through the Monitoring page again.

Service Health Panel

The top of the page shows tracked services and their last-known status (healthy, error, unknown). Updated in-memory each time the event consumer processes an event for a service. On app restart, the panel is rebuilt from the most recent alert per service in the database — so status survives restarts without needing to reprocess events.

Send Test Event (admin only)

The Send test event button in the Monitoring page header pushes a synthetic Lambda alarm to SQS. Requires event infrastructure to be set up. Shows an error toast if sqs_queue_url is not configured.

API

GET  /api/monitoring/alerts?limit=50   → list recent alerts
GET  /api/monitoring/alerts/{id}       → single alert detail
GET  /api/monitoring/services          → service health summary
GET  /api/monitoring/stream            → SSE stream of new alerts (real-time push)

The SSE stream (/api/monitoring/stream) sends a connected event on open, an alert event for each new alert as it is added, and a heartbeat every 20 seconds to keep the connection alive. The frontend subscribes on page load and falls back to 30-second polling if the connection drops.

Source Files

File	Purpose
`frontend/src/pages/MonitoringPage.tsx`	Live incident feed UI
`frontend/src/pages/AlertDetailPage.tsx`	Alert detail + investigate deeplink
`src/api/routers/monitoring.py`	REST endpoints + SSE stream
`src/agent/monitor_store.py`	In-process service status + DB-backed alert, notification, and incident claim persistence; SSE subscriber broadcast
`src/agent/investigation_runner.py`	Shared investigation runner used by both poller and event consumer
`src/agent/poller.py`	Proactive polling loop (bounded ThreadPoolExecutor, 4 workers)
`src/agent/event_consumer.py`	SQS long-poll consumer
`src/agent/event_infra.py`	Create/teardown SQS + EventBridge + alarm
`src/agent/context_collectors.py`	Per-event-type boto3 context enrichment
`scripts/test_pipeline.py`	Pipeline test script (direct SQS push or Lambda invoke)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Monitoring Dashboard

Detection Modes

Proactive Polling

What it does

Flow

When to use

Configuration

Deduplication

Source

Event-Driven Detection

What it does

Flow

When to use

Why 2–5 minutes for alarm-based events?

Event types and rules

Aggregate Lambda alarm

Setup

Context collectors

Source

Choosing Between the Two

Testing the Pipeline

Monitoring Dashboard UI

Incident Feed

Alert Detail Page

Session continuity and sidebar

Service Health Panel

Send Test Event (admin only)

API

Source Files

Related Documentation

FilesExpand file tree

monitoring.md

Latest commit

History

monitoring.md

File metadata and controls

Monitoring Dashboard

Detection Modes

Proactive Polling

What it does

Flow

When to use

Configuration

Deduplication

Source

Event-Driven Detection

What it does

Flow

When to use

Why 2–5 minutes for alarm-based events?

Event types and rules

Aggregate Lambda alarm

Setup

Context collectors

Source

Choosing Between the Two

Testing the Pipeline

Monitoring Dashboard UI

Incident Feed

Alert Detail Page

Session continuity and sidebar

Service Health Panel

Send Test Event (admin only)

API

Source Files

Related Documentation