"One service to watch them all, one service to find them, one service to alert them all, and in the Teams channel bind them."
Agentic AI post-deploy monitoring service. ESS watches production deployments in real time using Datadog, Sentry, and log search tools — orchestrated by an LLM reasoning loop — and escalates to MS Teams when issues are detected.
Current runtime note: the shipped path is Datadog-first Bedrock monitoring with
deterministic Pup fallback, release-aware Sentry follow-up for degraded
Sentry-enabled services, optional local debug artifacts under
_local_observability/, config-gated Teams delivery, and a checked-in
ess-harness CLI for repeatable live and degraded validation runs.
Log Scout and fuller multi-tool orchestration remain future work.
- GitLab pipeline completes a production deploy
- ESS receives a deploy trigger with service metadata
- Scheduler ticks run repeated health checks for a configurable window
- Bedrock tool loop calls Pup-backed Datadog tools and can deepen within a cycle
- If Datadog degrades, ESS performs release-aware Sentry follow-up for the affected Sentry-enabled services
- If the LLM path fails, deterministic Datadog triage still preserves the monitoring window
- If Teams is enabled, ESS posts warning, critical, and summary notifications
ESS does not remediate. It watches, investigates, and reports.
| Component | Technology |
|---|---|
| Runtime | Python 3.14+ |
| Package manager | uv |
| HTTP framework | FastAPI + uvicorn |
| LLM | AWS Bedrock Converse (current runtime: Claude Sonnet 4.6) |
| Datadog | Pup CLI (320+ commands, agent-mode JSON) |
| Sentry | REST API (self-hosted) |
| Log search | ESS Log Scout (remote agent on syslog servers) |
| Scheduler | APScheduler |
| Notifications | MS Teams incoming webhook (Adaptive Cards) |
- Deploy triggers, scheduler-driven monitoring windows, session APIs, and the Datadog Pup tool layer are live.
- The health-check runtime is Datadog-first and now adds release-aware Sentry follow-up using project details, release details, new release issue groups, and top issue details.
- Bedrock auth uses native
AWS_BEARER_TOKEN_BEDROCKsupport through botocore; ESS no longer decodes bearer tokens into raw AWS key/secret pairs. - The current agent runtime uses Claude Sonnet 4.6 for both triage and investigation turns.
- When
ESS_DEBUG_TRACE_ENABLED=true, ESS writes session-scoped traces and shared debug logs under_local_observability/. - The
ess-harnessCLI supportsliveruns against an existing local ESS instance anddegradedruns that force the Datadog-to-Sentry path with a temporary local server. - Log Scout, broader Bedrock-level orchestration, and Teams retry/backoff remain future work.
Use the checked-in harness when you want a repeatable local validation path without hand-driving curl calls and polling.
uv run ess-harness
uv run ess-harness live \
--trigger docs/examples/triggers/example-service-e2e.json
uv run ess-harness degraded \
--trigger _local_observability/triggers/pason-well-service-qa-degraded-e2e.jsonliveposts a trigger to an already running ESS instance and waits for the session to finish.degradedstarts a temporary local ESS server and injects deterministic degraded Datadog responses while keeping Bedrock and Sentry live.- Both commands write status and summary artifacts under
_local_observability/.
# Install dependencies
uv sync
# Configure
cp config/.env.example config/.env
# Edit config/.env with your credentials
# Run
uv run uvicorn src.main:app --host 0.0.0.0 --port 8080
# Test
uv run pytestSee docs/guides/GETTING_STARTED.md for full setup.
- Agent Operating Manual — for AI coding agents
- Documentation Index — navigation hub
- Architecture — system design
- Configuration — env vars reference
- Getting Started — setup guide
- Development — commands, workflows, and harness usage
- Trigger End-to-End Datadog Pup Integration — smoke runs, longer-window validation, and harness workflows