Skip to content
BranchesTags

Repository files navigation

Self-Healing Distributed System

A production-grade distributed system that detects service failures, makes intelligent severity-aware recovery decisions through AWS Lambda, and restores full health — without human intervention. Built around a config-driven gateway — adding a new service requires only a single JSON entry in services_config.json. No code changes, no redeployment of the system itself.

Python FastAPI Docker AWS Lambda EventBridge CloudWatch EC2 ECR S3 pydantic

Live Demo

Deployed on AWS EC2 — 54.224.134.71

Service URL Description
gateway — core-service http://54.224.134.71:8000/core-service Routes to core-service (strategy: fallback)
gateway — payment-service http://54.224.134.71:8000/payment-service Routes to payment-service (strategy: escalate)
gateway — movie-service http://54.224.134.71:8000/movie-service Routes to movie-service (strategy: fallback)
api-service health http://54.224.134.71:8000/health Gateway health check
fallback-service health http://54.224.134.71:8002/health Shared fallback backend
recovery-agent health http://54.224.134.71:8003/health Recovery executor 
# Normal responses (via generic gateway)
curl http://54.224.134.71:8000/core-service
curl http://54.224.134.71:8000/payment-service
curl http://54.224.134.71:8000/movie-service

# Trigger core-service crash → watch auto-recovery (restart strategy)
curl -X POST http://54.224.134.71:8001/fail
curl http://54.224.134.71:8000/core-service   # degraded=true (fallback active)
sleep 30
curl http://54.224.134.71:8000/core-service   # degraded=false (auto-recovered)

# Trigger payment-service crash → Lambda forces severity=HIGH (escalate strategy)
curl -X POST http://54.224.134.71:8010/fail
curl http://54.224.134.71:8000/payment-service  # HTTP 503 — no fallback

# Trigger movie-service crash → Lambda routes to fallback-service
curl -X POST http://54.224.134.71:8020/fail
curl http://54.224.134.71:8000/movie-service    # degraded=true (fallback active)

Note: The EC2 instance may be stopped to avoid AWS costs. If the above URLs are unreachable, refer to the Setup Instructions to run locally.

Table of Contents

Architecture Overview

flowchart TD
    client(["Client"])

    subgraph EC2["AWS EC2 — Docker Compose"]
        cfg[["services_config.json\nsingle source of truth"]]

        subgraph GW["api-service :8000 — Config-Driven Gateway"]
            sreg["ServiceRegistry\nloads config at startup"]
            gwsvc["GatewayService · GET /{service_name}\nper-service CircuitBreaker"]
        end

        core["core-service :8001\nstrategy: fallback"]
        fb["fallback-service :8002\nshared fallback backend"]
        pay["payment-service :8010\nstrategy: escalate · critical"]
        mov["movie-service :8020\nstrategy: fallback"]
        mon["monitor\nasyncio.gather() · 5s · parallel"]

        subgraph AGENT["recovery-agent :8003"]
            ra["hmac token check\ndocker restart / stop\ncapture logs (tail 200)\nwrite history (flock) · emit metrics"]
            cr["crash_reports/\nincidents/ · tests/"]
        end
    end

    subgraph AWS["AWS Cloud"]
        eb["EventBridge\nSelfHealingFailureRule"]

        subgraph LAM["Lambda — RecoveryHandler"]
            policy["SmartRecoveryPolicy\nLOW / MEDIUM / HIGH / CRITICAL"]
            strat["Strategy Override"]
            rollback["RollbackManager"]
        end

        dlq["SQS DLQ"]
        cw["CloudWatch\nMetrics + Dashboard · 20 widgets"]
        s3["S3 — self-healing-crash-reports\nincidents/<service>/<date>/"]
    end

    client -->|"GET /{service_name}"| gwsvc
    cfg -.->|"drives routing + strategies"| sreg
    sreg --> gwsvc
    gwsvc -->|"cb CLOSED · strategy=fallback"| core
    gwsvc -->|"cb CLOSED · strategy=escalate"| pay
    gwsvc -->|"cb CLOSED · strategy=fallback"| mov
    gwsvc -->|"cb OPEN · strategy=fallback"| fb
    cfg -.->|drives service list| mon
    cfg -.->|bundled in zip| LAM
    core -.->|health poll| mon
    fb -.->|health poll| mon
    pay -.->|health poll| mon
    mov -.->|health poll| mon
    gwsvc -->|"FallbackUsed + CircuitBreakerState"| cw
    mon -->|ServiceFailureDetected| eb
    mon -->|FailureDetectedCount| cw
    eb -->|invoke| LAM
    policy --> strat
    strat --> rollback
    LAM -->|"POST /action + X-Recovery-Token"| ra
    LAM -.->|on hard failure| dlq
    LAM -->|"severity + escalation metrics"| cw
    ra -->|"RecoverySuccess + Duration"| cw
    ra -->|"on enable_fallback\nsnapshot last 200 log lines"| cr
    cr -.->|"real incidents only"| s3
Loading

End-to-end recovery time: ~5–30 seconds from crash detection to full health.

Phase Progression

This project was built incrementally across 8 phases, each adding a production-grade capability:

Phase What Was Built
Phase 1 Core microservices — api-service, core-service, fallback-service running in Docker Compose
Phase 2 Health monitor polling every 5s, detecting crashes and latency degradation
Phase 3 AWS EventBridge integration — monitor publishes failure events; Lambda receives and logs them
Phase 4 Lambda executes recovery — calls recovery-agent via HTTP to docker restart the failed container
Phase 5 Production hardening — circuit breaker, CloudWatch metrics, SQS DLQ, event cooldown, EC2 deployment
Phase 6 Advanced self-healing — SmartRecoveryPolicy, IncidentSeverity, escalation logic, RollbackManager, new CloudWatch widgets
Phase 7 Multi-service platform — services_config.json registry, per-service strategies (restart/escalate/fallback), payment-service and movie-service added as real-world examples, 20-widget dashboard
Phase 8 Config-driven gateway — api-service becomes a true generic proxy (GET /{service_name}). One GenericServiceClient, one ServiceRegistry, one call() method. Adding a new service = 1 JSON line, zero code changes. Monitor health checks parallelised with asyncio.gather() — cycle time = slowest single check, not sum.
Phase 9 Crash report capture + production hardening — recovery-agent snapshots the last 200 container log lines on enable_fallback, writes them to incidents/ (real) or tests/ (synthetic) by reason prefix, and mirrors real reports to S3 (s3://…/incidents/<service>/<date>/) for AWS-console-accessible durable storage. CircuitBreaker is now thread-safe (RLock), JSONL writes use fcntl.flock so concurrent Lambda retries don't interleave, the recovery token uses hmac.compare_digest, and target_service is regex-validated as defence-in-depth.
Phase 10 Image registry & immutable deploys — all 6 service images are built linux/amd64, tagged with both a release tag (e.g. v1.0.0) and :latest, and pushed to private Amazon ECR repos under self-healing/<service>. EC2 pulls images via the IAM-attached AmazonEC2ContainerRegistryReadOnly policy — no registry credentials on disk. A docker-compose.ecr.yml override file replaces every build: block with image: ${ECR_REGISTRY}/self-healing/<svc>:${IMAGE_TAG:-latest}, so the running stack on EC2 no longer rebuilds from source — it pulls the exact, scanned, content-addressed image that was tested. A systemd unit re-runs deploy-from-ecr.sh on every boot for self-healing at the deployment layer too.

Features

Feature Phase Description
Auto Recovery 4 Lambda triggers docker restart on failed services. No manual intervention.
Circuit Breaker 5 Three-state machine (CLOSED → OPEN → HALF_OPEN). Stops hammering a failing service.
Fallback Handling 5 api-service routes traffic to fallback-service while core-service recovers.
Event Cooldown 5 60s deduplication window prevents Lambda from firing multiple times per incident.
Secure Recovery 5 X-Recovery-Token header + service allowlist protect recovery-agent from unauthorized calls.
CloudWatch Metrics 5 Custom metrics across all services. Real-time failure and recovery visibility.
Recovery Audit Log 5 Every action appended to recovery_history.jsonl — permanent append-only log.
SQS Dead-Letter Queue 5 Failed Lambda invocations captured for post-incident analysis.
EC2 Deployment 5 All Docker services deployed on AWS EC2, accessible over the internet.
SmartRecoveryPolicy 6 Decision engine mapping failure type + severity → correct recovery action.
IncidentSeverity 6 Four-tier classification: LOW / MEDIUM / HIGH / CRITICAL based on failure frequency.
Escalation Logic 6 Automatic escalation when failures exceed thresholds within sliding time windows.
Action Override 6 CRITICAL severity overrides restart → enable_fallback to stop the unstable container.
RollbackManager 6 Dry-run rollback recommendations logged when CRITICAL severity is reached.
Severity Metrics 6 Three new CloudWatch metrics: IncidentSeverityCount, EscalationCount, RollbackRecommendedCount.
Service Registry 7 services_config.json is the single source of truth — monitor, Lambda, and recovery-agent all load from it. Adding a new service requires zero code changes.
Per-Service Strategy 7 Each service declares its recovery strategy: restart, escalate, or fallback. Lambda applies the correct strategy automatically.
Escalate Strategy 7 Critical services with no fallback (payment-service) have every failure forced to minimum severity=HIGH. Operator intervention is always alerted.
Fallback Strategy 7 Services with a declared fallback (movie-service) log FALLBACK_AVAILABLE with the target service name on CRITICAL, enabling traffic routing.
Multi-Service Dashboard 7 20-widget CloudWatch dashboard with Phase 7 section: per-service failures, recovery outcomes, severity, escalations, rollback, and duration broken down by service.
Crash Report Capture 9 Before enable_fallback stops a container, recovery-agent captures docker logs --tail 200 and writes a structured incident report. Routed by reason prefix: [TEST]… → crash_reports/tests/, real Lambda calls → crash_reports/incidents/. Keeps the developer-facing folder uncluttered.
S3 Durable Crash Storage 9 Real incident reports are mirrored to S3 with date-partitioned keys (s3://<bucket>/incidents/<service>/<YYYY-MM-DD>/<file>.txt). Survives EC2 restart/replacement, browsable in the AWS console, and lifecycle-rule-friendly (Glacier after 90d). Test reports are skipped to keep the bucket clean.
Concurrency Hardening 9 CircuitBreaker state mutations protected by threading.RLock. JSONL audit-log appends serialized via fcntl.flock(LOCK_EX) so Lambda retries can't interleave records.
Defence-in-Depth 9 hmac.compare_digest for the recovery token (no timing oracle). target_service regex-validated at the schema layer (^[a-zA-Z0-9][a-zA-Z0-9_.-]*$) so a bad allowlist can't escape the crash-report directory. Pooled httpx.AsyncClient with follow_redirects=False and JSON-decode failures surfaced as request failures (engages the breaker, not a 500 leak).
Private Image Registry (ECR) 10 Six private ECR repos (self-healing/api-service, …/core-service, etc.) hold immutable, scanned images. Builds run --platform linux/amd64 so Apple Silicon laptops produce x86_64 images that boot on EC2. Each release is double-tagged (v1.2.3 + latest) so rollback is one re-pull away. scanOnPush=true runs CVE detection on every push.
Pull-Based Deploys 10 EC2 deploys via docker compose -f docker-compose.yml -f docker-compose.ecr.yml pull && up -d — no git pull, no docker build on the box. The override file uses build: !reset null so Compose can't accidentally fall back to building from source. ECR auth is fetched per-deploy via aws ecr get-login-password against the EC2 IAM role; no static credentials are stored. A systemd unit re-runs the deploy script on every boot, so an instance replacement self-heals without operator action.

Tech Stack

Layer Technology
Services Python 3.12, FastAPI, uvicorn
Packaging Docker, Docker Compose
Health Monitor Python (httpx async, boto3) — parallel health checks via asyncio.gather()
Service Registry services_config.json — single source of truth for monitor, gateway, Lambda, and recovery-agent
Event Bus AWS EventBridge
Serverless Recovery AWS Lambda (Python 3.12)
Decision Engine SmartRecoveryPolicy + strategy-aware overrides (Phase 7)
Infrastructure AWS EC2 (t2.micro, Ubuntu 22.04)
Image Registry AWS ECR — private repos under self-healing/*, scan-on-push, IAM-only auth
Durable Crash Storage AWS S3 — date-partitioned incident reports
Dead-Letter Queue AWS SQS
Observability AWS CloudWatch (custom metrics + 20-widget dashboard)
Deployment AWS Systems Manager (SSM) send-command
Configuration pydantic-settings (env-based)
Testing pytest, 74-test suite — 41 api-service unit tests (CircuitBreaker, ServiceRegistry, GatewayService) + 33 Lambda unit tests

System Workflow

Normal Operation

Client ──► GET /core-service ──► api-service gateway ──► core-service ──► { "source": "core-service", "degraded": false }
Client ──► GET /movie-service ──► api-service gateway ──► movie-service ──► { "source": "movie-service", "degraded": false }
Client ──► GET /payment-service ──► api-service gateway ──► payment-service ──► { "source": "payment-service", "degraded": false }

Failure → Auto-Recovery (8 steps)

Step 1   Client calls GET /core-service
         GatewayService looks up "core-service" in ServiceRegistry
         GenericServiceClient tries core-service → returns 503 (crashed)

Step 2   Circuit Breaker opens after 3 consecutive failures
         api-service routes all traffic to fallback-service
         → { "source": "fallback-service", "degraded": true }

Step 3   Monitor detects HTTP 503 on core-service (5-second poll)
         Publishes event to AWS EventBridge:
         { "source": "selfhealing.monitor",
           "detail-type": "ServiceFailureDetected",
           "detail": { "service_name": "core-service", "failure_type": "crash" } }

Step 4   EventBridge rule matches → invokes Lambda
         Event cooldown prevents duplicate invocations for 60 seconds

Step 5   Lambda: loads services_config.json → looks up core-service strategy = "restart"
         SmartRecoveryPolicy evaluates severity:
         failure_count_5min=1 → LOW  → action = restart_service
         failure_count_5min=3 → HIGH → action = restart_service (+ escalation)
         failure_count_10min≥5→ CRITICAL → action = enable_fallback (action override)

Step 6   Lambda calls POST /action on recovery-agent with:
         { "action": "restart_service", "target_service": "core-service",
           "severity": "LOW", "recovery_strategy": "standard_restart" }

Step 7   recovery-agent validates token + executes docker restart core-service
         Writes record to recovery_history.jsonl
         Emits CloudWatch metrics: RecoverySuccess, IncidentSeverityCount

Step 8   core-service restarts and passes health check
         Circuit breaker probes (HALF_OPEN) → success → CLOSED
         api-service resumes: { "source": "core-service", "degraded": false }
         Monitor clears cooldown. System fully healed.

Phase 7 — Strategy-Aware Multi-Service Recovery

The same pipeline applies to all registered services, but Lambda applies a different strategy depending on the service:

payment-service crash detected
       │
       ▼
Lambda: strategy = "escalate" (critical=true, no fallback)
       │
       ├── SmartRecoveryPolicy: severity=LOW (first failure)
       │
       └── Strategy override: LOW → HIGH (forced minimum for critical services)
           Logs: CRITICAL_SERVICE_NO_FALLBACK — operator intervention required
           Emits: EscalationCount metric with ServiceName=payment-service

movie-service crash detected
       │
       ▼
Lambda: strategy = "fallback", fallback_service = "fallback-service"
       │
       ├── SmartRecoveryPolicy: if severity=CRITICAL → action=enable_fallback
       │
       └── Logs: FALLBACK_AVAILABLE — traffic should route to fallback-service
           recovery-agent: docker stop movie-service (enable_fallback action)

Circuit Breaker State Machine

          3 failures                  30s timer expires
CLOSED ─────────────────► OPEN ──────────────────────────► HALF_OPEN
  ▲                                                              │
  │              probe succeeds (1 request passes)              │
  └──────────────────────────────────────────────────────────────┘
                        probe fails → back to OPEN

SmartRecoveryPolicy Decision Tree

Failure received
       │
       ├── failure_type = "slow"  ──────────────────────────► enable_fallback
       │                                                      (always, any severity)
       │
       └── failure_type = "crash" or "timeout"
                  │
                  ├── count_10min ≥ 5  → CRITICAL ──────────► enable_fallback
                  │                      is_escalated=True     (action override)
                  │
                  ├── count_5min  ≥ 3  → HIGH ───────────────► restart_service
                  │                      is_escalated=True     (+ escalation logged)
                  │
                  ├── count_5min  ≥ 2  → MEDIUM ─────────────► restart_service
                  │
                  └── count_5min  = 1  → LOW ───────────────► restart_service

Phase 7 overlay:
  strategy=escalate → minimum severity = HIGH regardless of failure count
  strategy=fallback  → on enable_fallback, logs FALLBACK_AVAILABLE with fallback service name

Key Concepts

Circuit Breaker

A safety switch between api-service and core-service that prevents cascading failures.

  • CLOSED — normal state. Every request passes through to core-service.
  • OPEN — tripped after 3 consecutive failures. All requests bypass core-service and go directly to fallback-service. This gives core-service time to recover without being hammered.
  • HALF_OPEN — after 30 seconds in OPEN, one test request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens for another 30 seconds.

This pattern is equivalent to Netflix Hystrix or Java's Resilience4j, implemented from scratch in Python.

Fallback

When core-service is unavailable (circuit OPEN or HTTP 503), api-service calls fallback-service instead. The response indicates degraded mode:

{ "source": "fallback-service", "degraded": true, "message": "Service temporarily unavailable" }

The client always gets a response — never a 503 error — while the system heals in the background.

Smart Recovery

The SmartRecoveryPolicy class inside Lambda replaces a static action map with severity-aware decision making. It uses module-level Python dicts that persist across warm Lambda invocations (no DynamoDB needed) to track failure history in 5-minute and 10-minute sliding windows.

1st crash in 5min  → LOW    → restart
2nd crash in 5min  → MEDIUM → restart
3rd crash in 5min  → HIGH   → restart + escalation alert
5th crash in 10min → CRITICAL → stop the container (enable_fallback)

At CRITICAL, the system recognizes that restarting a service that keeps crashing is futile — it stops the container and routes all traffic through fallback until a human intervenes.

Service Registry (Phase 7)

services_config.json is the single source of truth for all services. Every component loads from it at startup:

  • Monitor — dynamically builds the list of URLs to poll. Adding a new service to the registry immediately starts monitoring it — no code change needed.
  • Lambda — loads the registry at module init (survives warm invocations). Reads each service's strategy, fallback target, and criticality.
  • Recovery-agent — ALLOWED_SERVICES env var is driven by the registry, whitelisting which containers the agent is permitted to restart.
{
  "service_name": "payment-service",
  "health_url": "http://localhost:8010/health",
  "container_name": "payment-service",
  "strategy": "escalate",
  "fallback_service": null,
  "critical": true
}

Strategy-Aware Recovery (Phase 7)

Each service declares one of three strategies:

Strategy Behaviour Example service
restart Standard recovery. Severity ladder applies (LOW → CRITICAL). Falls back on CRITICAL. core-service
escalate Minimum severity = HIGH on every failure. For critical services with no fallback — operator must be notified every time. payment-service
fallback Same as restart, but logs FALLBACK_AVAILABLE with the named fallback service on CRITICAL. Enables traffic routing. movie-service

Crash Report Capture (Phase 9)

When SmartRecoveryPolicy escalates to CRITICAL and Lambda fires enable_fallback, the container is about to be stopped — its logs would be unreachable after docker stop (and gone entirely after docker rm). recovery-agent therefore snapshots the last 200 lines before stopping the container and writes a structured report:

========================================================================
CRASH REPORT — movie-service
Captured  : 2026-04-30T22-31-05-123456Z
Severity  : CRITICAL
Failures  : 5 consecutive crashes
Action    : enable_fallback (enable_fallback triggered)
Reason    : 5 crashes in 10 min — service unstable
Escalation: ESCALATION: movie-service failed 5x — switching to fallback
========================================================================

── CONTAINER LOGS (last 200 lines) ──
…

Routing by reason prefix keeps developer-facing storage uncluttered:

Reason starts with Local path S3 mirrored?
[TEST]… recovery-agent/data/crash_reports/tests/ No
anything else (real Lambda) recovery-agent/data/crash_reports/incidents/ Yes

S3 layout (when S3_CRASH_REPORTS_BUCKET is set):

s3://<bucket>/incidents/<service>/<YYYY-MM-DD>/<service>_<UTC>.txt
e.g. s3://self-healing-crash-reports/incidents/movie-service/2026-04-30/movie-service_2026-04-30T22-31-05-123456Z.txt

Date-partitioning is intentional: easy to find "today's incidents" in the console, plays well with S3 lifecycle rules (Glacier after 90 days), and Athena/S3 Select can query a date range cheaply. S3 upload errors are intentionally swallowed — a storage failure must never abort a recovery action that already worked.

Rollback Manager

The RollbackManager class tracks the last-known-good image tag for each service. When severity reaches CRITICAL, it logs a ROLLBACK_RECOMMENDED event with the image reference:

ROLLBACK_RECOMMENDED: service=core-service  image=core-service:latest  (dry-run)

The recommendation is logged and emitted as a CloudWatch metric but not executed — a human or automation pipeline can act on it.

API Reference

api-service — Port 8000 (Config-Driven Gateway)

Method Endpoint Description Response
GET /health Gateway health check {"status": "healthy", "service": "api-service"}
GET /{service_name} Route to any registered service. Strategy (fallback/escalate) applied automatically from config. Returns 404 for unknown names. {"source": "...", "result": {...}, "degraded": false}

Example calls:

curl http://localhost:8000/core-service       # strategy: fallback
curl http://localhost:8000/payment-service    # strategy: escalate — 503 when circuit OPEN
curl http://localhost:8000/movie-service      # strategy: fallback

core-service — Port 8001 (Primary Worker)

Method Endpoint Description Response
GET /health Returns 503 when crashed {"status": "healthy"/"unhealthy", "service": "core-service"}
GET /work Business logic endpoint called by api-service {"message": "...", "service": "core-service"}
GET /slow Simulates high latency response {"message": "...", "latency_simulated_seconds": 5.0}
POST /fail Test only — triggers a crash {"crashed": true}
POST /slow-mode Test only — activates slow mode on /work {"message": "slow mode enabled"}
POST /recover Manually resets all failure flags {"crashed": false}

fallback-service — Port 8002 (Degraded Mode)

Method Endpoint Description Response
GET /health Service health check {"status": "healthy", "service": "fallback-service"}
GET /fallback Returns degraded response (called by api-service when circuit is OPEN) {"message": "...", "degraded": true}

recovery-agent — Port 8003 (Recovery Executor)

Method Endpoint Auth Description
GET /health None Health check (token-free for load balancers)
POST /action X-Recovery-Token header required Executes Docker recovery command

POST /action request body:

{
  "action": "restart_service | enable_fallback | disable_fallback",
  "target_service": "core-service",
  "reason": "crash detected",
  "severity": "LOW | MEDIUM | HIGH | CRITICAL",
  "recovery_strategy": "standard_restart | fallback_on_critical",
  "failure_count": 1,
  "escalation_reason": null
}

POST /action response:

{
  "success": true,
  "action": "restart_service",
  "target_service": "core-service",
  "message": "Container 'core-service' restarted successfully.",
  "timestamp": "2026-04-28T10:00:00.000000+00:00",
  "command_result": {
    "success": true,
    "stdout": "core-service",
    "stderr": "",
    "returncode": 0
  }
}

payment-service — Port 8010 (Critical Service — Phase 7 Example)

Strategy: escalate — critical=true, no fallback. Every failure triggers minimum severity=HIGH.

Method Endpoint Description Response
GET /health Returns 503 when crashed {"status": "healthy"/"unhealthy", "service": "payment-service"}
GET /process-payment Simulated payment processing {"message": "Payment processed", "service": "payment-service"}
POST /fail Test only — triggers a crash {"crashed": true}
POST /recover Manually resets failure flag {"crashed": false}

movie-service — Port 8020 (Fallback Strategy — Phase 7 Example)

Strategy: fallback — on CRITICAL, Lambda logs FALLBACK_AVAILABLE and routes to fallback-service.

Method Endpoint Description Response
GET /health Returns 503 when crashed {"status": "healthy"/"unhealthy", "service": "movie-service"}
GET /catalog Returns movie catalog {"movies": [...], "service": "movie-service"}
POST /fail Test only — triggers a crash {"crashed": true}
POST /recover Manually resets failure flag {"crashed": false}

Project Structure

self-healing-system/
│
├── services_config.json        # Single source of truth — drives everything
│                               # monitor URLs · gateway routing · Lambda strategy · recovery allowlist
│
│── ─── SYSTEM COMPONENTS ──────────────────────────────────────────────────
│
├── api-service/                # Config-driven gateway (:8000)
│   └── app/
│       ├── clients/
│       │   └── generic_client.py       # One HTTP client for every service
│       ├── services/
│       │   ├── service_registry.py     # Loads services_config.json, creates circuit breakers
│       │   ├── gateway_service.py      # call(service_name) — fallback or escalate
│       │   └── circuit_breaker.py      # CLOSED/OPEN/HALF_OPEN state machine (per service)
│       └── routes/api_routes.py        # GET /health · GET /{service_name}
│
├── fallback-service/           # Degraded-mode backend (:8002)
│   └── app/routes/             # GET /fallback — always returns degraded response
│
├── recovery-agent/             # Docker command executor (:8003)
│   └── app/
│       ├── services/
│       │   ├── recovery_service.py     # restart / stop / start · crash report capture
│       │   ├── recovery_history.py     # append-only JSONL audit log (fcntl-locked)
│       │   └── docker_executor.py      # docker CLI wrapper · capture_logs(tail=200)
│       ├── publishers/
│       │   ├── cloudwatch_publisher.py # custom metrics (no-op when disabled)
│       │   └── s3_crash_report_publisher.py  # mirrors real reports to S3
│       └── routes/recovery_routes.py  # POST /action (hmac.compare_digest token)
│   └── data/
│       └── crash_reports/
│           ├── incidents/              # real Lambda-driven crashes (committed: ignored)
│           └── tests/                  # synthetic [TEST] reports (committed: ignored)
│
├── monitor/                    # Async parallel health watcher (runs on EC2, not Docker)
│   ├── monitor.py              # asyncio.run(monitor.run_async()) entry point
│   └── app/
│       ├── checkers/
│       │   ├── health_checker.py       # check_async() via httpx — parallel via gather()
│       │   └── latency_checker.py      # SLOW / VERY_SLOW classification
│       └── services/
│           ├── monitor_service.py      # run_check_cycle_async() — all checks fire at once
│           └── event_cooldown.py       # 60s deduplication window
│
├── aws/
│   ├── lambda/
│   │   ├── recovery_handler.py         # Strategy-aware, reads services_config.json
│   │   ├── smart_recovery_policy.py    # LOW/MEDIUM/HIGH/CRITICAL decision engine
│   │   ├── rollback_manager.py         # Dry-run rollback recommendations
│   │   └── tests/                      # 33 unit tests
│   ├── cloudwatch/
│   │   ├── dashboard.json              # 20-widget dashboard
│   │   └── create_dashboard.sh
│   └── setup/
│       ├── eventbridge_rule.json
│       └── iam_policy.json
│
│── ─── DEMO SERVICES (examples of services that plug into the system) ──────
│
├── demo-services/
│   ├── core-service/           # Demo: strategy=fallback (:8001)  GET /work
│   ├── payment-service/        # Demo: strategy=escalate (:8010)  GET /process-payment
│   └── movie-service/          # Demo: strategy=fallback (:8020)  GET /catalog
│
│── ─── OTHER ──────────────────────────────────────────────────────────────
│
├── demo-ui/index.html          # Single-file interactive demo (simulation)
├── docker-compose.yml          # Local dev: builds images from source
├── docker-compose.ecr.yml      # Phase 10: production override, pulls from ECR (build:!reset)
├── scripts/
│   ├── push-to-ecr.sh          # Phase 10: build linux/amd64 + push 6 images, double-tagged
│   └── deploy-from-ecr.sh      # Phase 10: aws ecr login → compose pull → up -d (runs on EC2)
├── tests/scripts/
└── docs/

Adding a new service: add one JSON block to services_config.json. The gateway, monitor, Lambda, and recovery-agent all pick it up automatically — no code changes anywhere.

Setup Instructions

Prerequisites

Tool Version Check
Docker + Docker Compose v2+ docker compose version
Python 3.10+ python3 --version
AWS CLI v2 aws --version
AWS account — Access key + secret configured

AWS Resources Required

Resource Name
Lambda Function SelfHealingRecoveryHandler
Lambda IAM Role Needs: lambda:InvokeFunction, cloudwatch:PutMetricData
EventBridge Rule SelfHealingFailureRule
SQS Dead-Letter Queue SelfHealingLambdaDLQ
CloudWatch Dashboard SelfHealingSystemDashboard
EC2 Instance Ubuntu 22.04, ports 8000–8003, 8010, 8020 open
S3 Bucket (optional, Phase 9) self-healing-crash-reports — durable storage for incident reports. EC2 IAM role needs s3:PutObject on <bucket>/incidents/*.
ECR Repositories (Phase 10) Six repos under self-healing/* — api-service, core-service, fallback-service, recovery-agent, payment-service, movie-service. EC2 IAM role needs AmazonEC2ContainerRegistryReadOnly.

Option A — Local (Docker Compose, no AWS required)

The fastest way to demo. Everything runs on your laptop — circuit breaker, fallback, recovery-agent, monitor — and you can trigger a crash and watch the system heal end-to-end without any AWS account.

# 1. Clone
git clone https://github.com/agrawal-2005/self-healing-system.git
cd self-healing-system

# 2. Bring up all 6 services (core, fallback, api, recovery-agent, payment, movie)
docker compose up --build -d
docker compose ps           # wait until every line shows "(healthy)"

# 3. Smoke test — gateway should hit core-service
curl -s http://localhost:8000/core-service | jq
# { "source": "core-service", "degraded": false, "result": { ... } }

# 4. Start the host-side monitor (parallel async health checks every 5s)
cd monitor
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Local-only demo: AWS credentials NOT required. CloudWatch publishing
# is a no-op when CLOUDWATCH_ENABLED=false.
CLOUDWATCH_ENABLED=false python3 monitor.py > /tmp/monitor.log 2>&1 &
cd ..

# 5. Trigger a crash on core-service and watch fallback engage
curl -X POST http://localhost:8001/fail
curl -s http://localhost:8000/core-service | jq
# { "source": "fallback-service", "degraded": true }   ← circuit OPEN

# 6. Recover manually (no Lambda needed for the local demo)
curl -X POST http://localhost:8001/recover
sleep 35                                        # wait for HALF_OPEN probe
curl -s http://localhost:8000/core-service | jq
# { "source": "core-service", "degraded": false }      ← circuit CLOSED again

# 7. Tail the monitor to see what it saw
tail -n 20 /tmp/monitor.log

# 8. Stop everything
pkill -f 'python3 monitor.py' || true
docker compose down

AWS-aware local mode: Want EventBridge → Lambda → recovery-agent firing during the local demo? Set AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_DEFAULT_REGION in monitor/.env, drop CLOUDWATCH_ENABLED=false, and follow the Lambda deploy steps in Option B. recovery-agent must be reachable from Lambda — easiest is ngrok http 8003 and point RECOVERY_AGENT_URL at the ngrok URL.

Option B — EC2 Deployment (Production)

All services run on an EC2 instance. Deployment uses AWS Systems Manager (SSM) — no SSH keys required.

# 1. Set your EC2 instance ID
INSTANCE_ID="i-xxxxxxxxxxxxxxxxx"
EC2="<your-ec2-public-ip>"

# 2. Deploy latest code via SSM
aws ssm send-command \
  --instance-ids "$INSTANCE_ID" \
  --document-name "AWS-RunShellScript" \
  --parameters "{\"commands\":[
    \"chown -R ubuntu:ubuntu /home/ubuntu/self-healing-system\",
    \"sudo -u ubuntu bash -c 'cd /home/ubuntu/self-healing-system && git pull origin main'\",
    \"docker compose up --build -d --wait\"
  ]}" \
  --region us-east-1

# 3. Verify all 6 services are healthy
curl http://$EC2:8000/health    # api-service
curl http://$EC2:8001/health    # core-service
curl http://$EC2:8002/health    # fallback-service
curl http://$EC2:8003/health    # recovery-agent
curl http://$EC2:8010/health    # payment-service
curl http://$EC2:8020/health    # movie-service

# 4. Restart monitor to pick up latest services_config.json
aws ssm send-command \
  --instance-ids "$INSTANCE_ID" \
  --document-name "AWS-RunShellScript" \
  --parameters "{\"commands\":[
    \"pkill -f 'python3 monitor.py' || true\",
    \"sudo -u ubuntu bash -c 'cd /home/ubuntu/self-healing-system/monitor && source .venv/bin/activate && nohup python3 monitor.py > /home/ubuntu/monitor.log 2>&1 &'\"
  ]}" \
  --region us-east-1

# 5. Deploy Lambda pointing to EC2
aws lambda update-function-configuration \
  --function-name SelfHealingRecoveryHandler \
  --environment "Variables={
    RECOVERY_AGENT_URL=http://$EC2:8003,
    TARGET_SERVICE=core-service,
    RECOVERY_TOKEN=dev-token,
    MAX_RETRIES=3,
    CLOUDWATCH_ENABLED=true,
    CLOUDWATCH_NAMESPACE=SelfHealingSystem
  }" \
  --region us-east-1

Option C — Pull from ECR instead of building on EC2 (Phase 10)

Once the project's images are stored in ECR (one-time setup below), EC2 no longer needs git pull or local Docker builds — it just docker compose pulls the exact image that was tested on your laptop or in CI.

Current state of this account's ECR: the 6 repos under self-healing/* exist but are empty (images were deleted to keep the AWS bill at $0 between demos). Run ./scripts/push-to-ecr.sh v1.0.0 to repopulate them — the create-repo step is already done.

One-time AWS setup

# 1. Create the 6 private repos
REGION=us-east-1
for repo in api-service core-service fallback-service recovery-agent payment-service movie-service; do
  aws ecr create-repository \
    --repository-name self-healing/$repo \
    --region $REGION \
    --image-scanning-configuration scanOnPush=true \
    --image-tag-mutability MUTABLE
done

# 2. Allow the EC2 instance role to pull from ECR
aws iam attach-role-policy \
  --role-name SelfHealingEC2Role \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly

Build & push (laptop or CI)

# Builds linux/amd64 (so it runs on x86_64 EC2 even from Apple Silicon),
# tags with the version arg + :latest, and pushes all 6 in one go.
./scripts/push-to-ecr.sh v1.0.0

What the script does, per service:

docker build --platform linux/amd64 \
  -t <acct>.dkr.ecr.us-east-1.amazonaws.com/self-healing/<svc>:v1.0.0 \
  -t <acct>.dkr.ecr.us-east-1.amazonaws.com/self-healing/<svc>:latest \
  <build-context>
docker push  <acct>.dkr.ecr.us-east-1.amazonaws.com/self-healing/<svc>:v1.0.0
docker push  <acct>.dkr.ecr.us-east-1.amazonaws.com/self-healing/<svc>:latest

Deploy on EC2 (pull, don't build)

# On EC2 (or via SSM send-command):
cd /home/ubuntu/self-healing-system
./scripts/deploy-from-ecr.sh v1.0.0      # or no arg → :latest

# Internally:
#   aws ecr get-login-password | docker login …
#   docker compose -f docker-compose.yml -f docker-compose.ecr.yml pull
#   docker compose -f docker-compose.yml -f docker-compose.ecr.yml up -d --remove-orphans
#   docker image prune -f

The override file docker-compose.ecr.yml is what flips every service from build: to image::

services:
  api-service:
    build: !reset null     # explicitly clear the inherited build: block (Compose v2.24+)
    image: ${ECR_REGISTRY}/self-healing/api-service:${IMAGE_TAG:-latest}
  # … same pattern for the other 5 services

Auto-pull on every EC2 boot (systemd)

sudo tee /etc/systemd/system/self-healing.service > /dev/null <<'EOF'
[Unit]
Description=Self-Healing System (pull from ECR)
After=docker.service network-online.target
Requires=docker.service

[Service]
Type=oneshot
RemainAfterExit=yes
User=ubuntu
WorkingDirectory=/home/ubuntu/self-healing-system
ExecStart=/home/ubuntu/self-healing-system/scripts/deploy-from-ecr.sh latest
ExecStop=/usr/bin/docker compose -f docker-compose.yml -f docker-compose.ecr.yml down

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now self-healing.service

After this, an instance restart (or replacement) re-pulls the latest images and brings the stack up — extending the project's self-healing posture down to the deployment layer itself.

Why pull-based ECR over git pull && docker build

git pull && docker build (Options A/B) docker compose pull from ECR (Option C)
Image rebuilt on EC2 every deploy — risk of "works on laptop, broken on prod" if base image, system libs, or Python wheels drift EC2 pulls the exact image bytes that were built and tested on the laptop / CI
EC2 needs source code, build toolchain, and network access to pip / apt mirrors EC2 only needs Docker + ECR pull rights — smaller attack surface
No image scanning scanOnPush=true runs Inspector CVE scan on every push
Rollback = git checkout <old-sha> && docker compose up --build (slow, may not be byte-identical) Rollback = IMAGE_TAG=v0.9.0 ./scripts/deploy-from-ecr.sh (~seconds)
Bandwidth-heavy on each deploy (full git clone of source + Python wheel downloads) Layer-cached pulls — usually only the changed layer is fetched

Testing the System

Unit Tests

# api-service — 41 tests (CircuitBreaker, ServiceRegistry, GatewayService)
cd api-service
pip install -r requirements.txt -r requirements-test.txt
pytest
# 41 passed

# Lambda — 33 tests (SmartRecoveryPolicy, RollbackManager)
cd aws/lambda
python -m pytest tests/ -v
# 33 passed

Gateway Routes Smoke Test (Phase 8)

Verifies all 6 service health checks, all GET /{service_name} routes, 404 for unknown services, circuit breaker isolation, and payment escalate strategy — no AWS required.

# Against local
chmod +x tests/scripts/gateway_routes_smoke_test.sh
./tests/scripts/gateway_routes_smoke_test.sh

# Against EC2
API_URL=http://54.224.134.71:8000 \
CORE_URL=http://54.224.134.71:8001 \
FALLBACK_URL=http://54.224.134.71:8002 \
RECOVERY_AGENT_URL=http://54.224.134.71:8003 \
PAYMENT_URL=http://54.224.134.71:8010 \
MOVIE_URL=http://54.224.134.71:8020 \
./tests/scripts/gateway_routes_smoke_test.sh

End-to-End Chaos Test

# Against local
chmod +x tests/scripts/critical_core_failure_recovery.sh
./tests/scripts/critical_core_failure_recovery.sh

# Against EC2
API_URL=http://54.224.134.71:8000 \
CORE_URL=http://54.224.134.71:8001 \
./tests/scripts/critical_core_failure_recovery.sh

Manual Tests

Test 1 — Verify baseline

curl http://$EC2:8000/core-service
# { "source": "core-service", "degraded": false }

Test 2 — Trigger a crash (restart strategy)

curl -X POST http://$EC2:8001/fail
# { "crashed": true }

Test 3 — Observe fallback immediately

curl http://$EC2:8000/core-service
# { "source": "fallback-service", "degraded": true }

Test 4 — Confirm full recovery

sleep 30
curl http://$EC2:8000/core-service
# { "source": "core-service", "degraded": false }

Test 5 — Phase 6 escalation demo (5 crashes, 65s apart)

for i in 1 2 3 4 5; do
  curl -s -X POST http://$EC2:8001/fail
  echo "Crash $i triggered"
  sleep 65
done
# Crash 5 → CRITICAL severity → enable_fallback + ROLLBACK_RECOMMENDED

Test 6 — Phase 7: payment-service escalate strategy

# First failure on payment-service — Lambda forces HIGH (not LOW)
curl -X POST http://$EC2:8010/fail

# Check Lambda logs for: CRITICAL_SERVICE_NO_FALLBACK
# Severity will be HIGH on the very first failure (escalate strategy)

Test 7 — Phase 7: movie-service fallback strategy

# Trigger movie-service CRITICAL (5 crashes)
for i in 1 2 3 4 5; do
  curl -s -X POST http://$EC2:8020/fail
  echo "Movie crash $i"
  sleep 65
done
# On CRITICAL → Lambda logs: FALLBACK_AVAILABLE: movie-service → fallback-service

Test 8 — Phase 9: crash report capture (direct curl)

Hits recovery-agent directly with a [TEST]-prefixed reason so the report lands in tests/ (not incidents/) and S3 upload is skipped:

curl -s -X POST http://$EC2:8003/action \
  -H "Content-Type: application/json" \
  -H "X-Recovery-Token: dev-token" \
  -d '{
    "action": "enable_fallback",
    "target_service": "movie-service",
    "reason": "[TEST] crash report capture verification",
    "severity": "CRITICAL",
    "failure_count": 5
  }' | jq

# Then confirm a report appeared
ls recovery-agent/data/crash_reports/tests/
# movie-service_2026-04-30T22-31-05-123456Z.txt

To restore movie-service after the test:

curl -s -X POST http://$EC2:8003/action \
  -H "Content-Type: application/json" \
  -H "X-Recovery-Token: dev-token" \
  -d '{"action":"disable_fallback","target_service":"movie-service","reason":"[TEST] cleanup"}'

Test 9 — Phase 7: simultaneous multi-service failure

# Crash all three services at the same time
curl -X POST http://$EC2:8001/fail &   # core-service
curl -X POST http://$EC2:8010/fail &   # payment-service
curl -X POST http://$EC2:8020/fail &   # movie-service
wait

# Each service triggers a separate EventBridge event → separate Lambda invocation
# Lambda applies: restart (core), escalate→HIGH (payment), fallback (movie)
# All three recover independently

CloudWatch Dashboard

Dashboard name: SelfHealingSystemDashboard
Namespace: SelfHealingSystem

Live Dashboard Screenshots

Failure Detection & Recovery Outcomes Dashboard Overview

Real-time view of crash detection (red) and automated recovery (green) across a 3-hour test window. Every failure spike on the left has a matching success spike on the right — the system self-healed every single time.

Recovery Pipeline Metrics Recovery Pipeline

Recovery duration peaked at 17.3s during load testing, fallback activated up to 8 times, and the circuit breaker stayed OPEN while the service was unstable. Lambda DLQ shows no data — Lambda never hard-failed across the entire test.

Incident Intelligence — Phase 6 Incident Intelligence

SmartRecoveryPolicy in action: CRITICAL escalations firing above the alert threshold, and rollback recommendations triggered above the rollback threshold. All 3 widgets confirm the severity escalation ladder (LOW → MEDIUM → HIGH → CRITICAL) worked as designed.

Service Recovery Strategies Service Recovery Strategies

Recovery duration breakdown per service alongside the strategy reference table. Each service carries a distinct recovery strategy: core-service restarts automatically (peaking at 1.7s), payment-service always escalates to minimum HIGH severity with no fallback, and movie-service routes to fallback-service on CRITICAL. The chart confirms only core-service has active recovery events — payment-service and movie-service lines appear when those services are exercised.

All 20 Widgets Explained

Phase 1–5 Widgets

Widget Metric What It Shows
Failure Detected FailureDetectedCount Each spike = one crash/timeout detected by monitor and sent to EventBridge. Dimension: failure_type.
Recovery Outcomes RecoverySuccessCount / RecoveryFailureCount Green = successful docker restart. Red = docker command failed.
Average Recovery Duration RecoveryDurationMs Time in ms from Lambda invocation to docker command completing.
Fallback Used FallbackUsedCount Each dot = one request served by fallback-service instead of core-service.
Circuit Breaker Opened CircuitBreakerOpenCount Each dot = circuit transitioned to OPEN. Frequent openings = persistent instability.
Circuit Breaker State CircuitBreakerState Gauge: 0=CLOSED (healthy), 1=HALF_OPEN (probing), 2=OPEN (all traffic on fallback).
Lambda DLQ SQS ApproximateNumberOfMessagesVisible Messages = Lambda hard-crashed. Zero = Lambda always handled events successfully.

Phase 6 Widgets

Widget Metric What It Shows
Incident Severity Distribution IncidentSeverityCount Lines per severity tier (LOW/MEDIUM/HIGH/CRITICAL). Shows whether failures are isolated or a pattern.
Escalation Count EscalationCount Published only for HIGH and CRITICAL severity. Spikes = system identified a persistent incident.
Rollback Recommended RollbackRecommendedCount Published when CRITICAL + baseline exists. Each count = one rollback recommendation logged.

Phase 7 Widgets (Multi-Service)

Widget What It Shows
Failures Detected by Service All 3 services on one chart — core-service, payment-service, movie-service. Isolates which service is failing.
Recovery Outcomes by Service Success/failure lines per service. Compare recovery rates across strategies.
Incident Severity by Service Severity breakdown per service. payment-service should only show HIGH/CRITICAL due to escalate strategy.
Escalations by Service payment-service appears on every single failure it has. core-service only appears after 3+ failures.
Rollback Recommended by Service Per-service rollback triggers. payment-service rollbacks are more frequent due to forced HIGH severity.
Recovery Duration by Service avg + p99 per service. Compares restart speed vs enable_fallback speed across strategies.
Strategy Reference Static table — core=restart, payment=escalate (critical, no fallback), movie=fallback (→ fallback-service).

Custom Metrics Reference

Metric Emitted By Dimensions
FailureDetectedCount monitor ServiceName, FailureType
FallbackUsedCount api-service ServiceName
CircuitBreakerOpenCount api-service ServiceName
CircuitBreakerState api-service ServiceName
RecoverySuccessCount recovery-agent ServiceName, TargetService, Action
RecoveryFailureCount recovery-agent ServiceName, TargetService, Action
RecoveryDurationMs recovery-agent ServiceName, TargetService, Action
IncidentSeverityCount Lambda ServiceName, Severity
EscalationCount Lambda ServiceName, Severity
RollbackRecommendedCount Lambda ServiceName

Suggested Alarms

# Alert if any recovery failure occurs
aws cloudwatch put-metric-alarm \
  --alarm-name "RecoveryFailed" \
  --namespace SelfHealingSystem \
  --metric-name RecoveryFailureCount \
  --statistic Sum --period 300 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions <YOUR_SNS_ARN> \
  --region us-east-1

# Alert on CRITICAL escalation for any service
aws cloudwatch put-metric-alarm \
  --alarm-name "CriticalEscalation" \
  --namespace SelfHealingSystem \
  --metric-name EscalationCount \
  --dimensions Name=Severity,Value=CRITICAL \
  --statistic Sum --period 300 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions <YOUR_SNS_ARN> \
  --region us-east-1

# Alert specifically when payment-service escalates (critical service)
aws cloudwatch put-metric-alarm \
  --alarm-name "PaymentServiceEscalation" \
  --namespace SelfHealingSystem \
  --metric-name EscalationCount \
  --dimensions Name=ServiceName,Value=payment-service Name=Severity,Value=HIGH \
  --statistic Sum --period 60 --threshold 1 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --evaluation-periods 1 \
  --alarm-actions <YOUR_SNS_ARN> \
  --region us-east-1

Sample Outputs

Monitor log — Phase 7 multi-service healthy

2026-04-28 10:01:57 [monitor] INFO: [OK] core-service      status=UP   latency=2.9ms
2026-04-28 10:01:57 [monitor] INFO: [OK] payment-service   status=UP   latency=2.9ms
2026-04-28 10:01:57 [monitor] INFO: [OK] movie-service     status=UP   latency=2.7ms
2026-04-28 10:01:57 [monitor] INFO: All 3/3 services healthy

Monitor log — during a recovery cycle

2026-04-28 10:02:02 [monitor] ERROR: HealthChecker [core-service]: DOWN — Connection refused
2026-04-28 10:02:02 [monitor] INFO:  EventBridgePublisher: published event service=core-service failure=crash
2026-04-28 10:02:08 [monitor] INFO:  EventCooldown: cleared 1 timer(s) for 'core-service' on recovery
2026-04-28 10:02:08 [monitor] INFO:  [OK] core-service     status=UP   latency=3.4ms
2026-04-28 10:02:08 [monitor] INFO:  All 3/3 services healthy

Lambda log — Phase 7 payment-service (escalate strategy)

STRATEGY: service=payment-service  strategy=escalate  fallback=None  critical=True
SmartRecoveryPolicy: action=restart_service  severity=LOW  count_5m=1  escalated=False
CRITICAL_SERVICE_NO_FALLBACK: payment-service has no fallback — upgrading severity LOW → HIGH
lambda_handler: complete — service=payment-service  strategy=escalate  success=True
  severity=HIGH  duration=312ms  attempts=1  escalated=True  rollback=False

Lambda log — Phase 7 movie-service (fallback strategy, CRITICAL)

STRATEGY: service=movie-service  strategy=fallback  fallback=fallback-service  critical=False
SmartRecoveryPolicy: action=enable_fallback  severity=CRITICAL  count_5m=4  count_10m=5
FALLBACK_AVAILABLE: movie-service is CRITICAL — traffic should route to fallback-service
lambda_handler: complete — service=movie-service  strategy=fallback  success=True
  severity=CRITICAL  duration=287ms  attempts=1  escalated=True  rollback=True

Lambda log — Phase 6 CRITICAL escalation

SmartRecoveryPolicy: action=enable_fallback  severity=CRITICAL  strategy=fallback_on_critical  count_5m=4  count_10m=5
ESCALATION: core-service failed 5x in 10 min — service is critically unstable — switching to enable_fallback
ROLLBACK_RECOMMENDED: service=core-service  image=core-service:latest  (dry-run)
lambda_handler: complete — success=True  severity=CRITICAL  duration=241ms  attempts=1  escalated=True  rollback=True

recovery-agent log

2026-04-28 10:00:06 [recovery_service] INFO: action=restart_service  target=core-service  severity=LOW
2026-04-28 10:00:06 [docker_executor]  INFO: running docker restart core-service
2026-04-28 10:00:07 [recovery_history] INFO: recorded action=restart_service  service=core-service  success=True  duration=520ms

Recovery history record (recovery_history.jsonl)

{
  "timestamp": "2026-04-28T10:00:06.361801+00:00",
  "service_name": "core-service",
  "failure_type": "crash",
  "action": "restart_service",
  "success": true,
  "message": "Container 'core-service' restarted successfully.",
  "recovery_duration_ms": 520.34,
  "returncode": 0,
  "severity": "LOW",
  "failure_count": 1,
  "recovery_strategy": "standard_restart",
  "escalation_reason": null
}

Phase 7 payment-service record

{
  "timestamp": "2026-04-28T10:05:00.000000+00:00",
  "service_name": "payment-service",
  "failure_type": "crash",
  "action": "restart_service",
  "success": true,
  "message": "Container 'payment-service' restarted successfully.",
  "recovery_duration_ms": 312.45,
  "returncode": 0,
  "severity": "HIGH",
  "failure_count": 1,
  "recovery_strategy": "standard_restart",
  "escalation_reason": "ESCALATION: payment-service is a critical service with no fallback — severity forced to HIGH on every failure"
}

Crash report file (Phase 9)

Saved to recovery-agent/data/crash_reports/incidents/<service>_<UTC>.txt and mirrored to S3 when the bucket is configured:

========================================================================
CRASH REPORT — movie-service
Captured  : 2026-04-30T22-31-05-123456Z
Severity  : CRITICAL
Failures  : 5 consecutive crashes
Action    : enable_fallback (enable_fallback triggered)
Reason    : 5 crashes in 10 min — service unstable
Escalation: ESCALATION: movie-service failed 5x — switching to fallback
========================================================================

── CONTAINER LOGS (last 200 lines) ──

INFO:     Started server process [1]
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8020
INFO:     127.0.0.1:54312 - "GET /catalog HTTP/1.1" 200 OK
ERROR:    movie-service: simulated crash triggered via /fail
ERROR:    movie-service: /catalog returning 503 (crashed=true)
…

── END OF REPORT ──

2026-04-30 22:31:05 [recovery_service] INFO: crash report saved → /app/data/crash_reports/incidents/movie-service_2026-04-30T22-31-05-123456Z.txt
2026-04-30 22:31:05 [s3_crash_report_publisher] INFO: uploaded crash report → s3://self-healing-crash-reports/incidents/movie-service/2026-04-30/movie-service_2026-04-30T22-31-05-123456Z.txt
2026-04-30 22:31:05 [recovery_service] INFO: crash report mirrored to S3 → s3://self-healing-crash-reports/incidents/movie-service/2026-04-30/movie-service_2026-04-30T22-31-05-123456Z.txt

Circuit breaker log progression

WARNING: CircuitBreaker: failure recorded 1/3
WARNING: CircuitBreaker: failure recorded 2/3
WARNING: CircuitBreaker: CLOSED → OPEN (failures=3/3 threshold=3)
INFO:    FALLBACK_TRIGGERED: source=circuit_open
INFO:    CloudWatch: CircuitBreakerOpenCount+1  CircuitBreakerState=2
WARNING: CircuitBreaker: OPEN — blocking core-service call (29s until probe)
INFO:    CircuitBreaker: OPEN → HALF_OPEN after 30s cooldown
INFO:    CircuitBreaker: HALF_OPEN → CLOSED (core-service recovered)

Chaos test result

══════════════════════════════════════════════════════════════════
  TEST SUMMARY: critical_core_failure_recovery
══════════════════════════════════════════════════════════════════

  Total steps checked : 31
  Passed              : 31
  Failed              : 0

  ██████████████████████████████████████████
    RESULT: PASS — All steps completed
    Self-healing pipeline is working
  ██████████████████████████████████████████

Future Improvements

Improvement Description
ECS / Fargate deployment Replace Docker Compose with ECS tasks. Remove direct EC2 access — recovery-agent runs inside the same VPC as Lambda, eliminating token-over-HTTP security concerns.
Auto rollback execution Phase 7 logs rollback recommendations (dry-run). A future phase would execute docker pull <previous-tag> + container replacement when recovery fails after N retries.
AI-based anomaly detection Replace threshold-based latency checks with a time-series model (e.g. AWS Lookout for Metrics) that detects statistical anomalies without fixed thresholds.
Multi-region failover Replicate the EventBridge → Lambda → recovery pipeline across two regions. Promote secondary on primary failure.
Slack / PagerDuty alerts Wire CloudWatch alarms → SNS → Lambda → Slack webhook. On-call engineers receive a message with recovery status and severity within seconds of detection. Especially important for escalate strategy services.
DynamoDB failure history Replace module-level in-memory failure tracking with DynamoDB for persistence across cold Lambda starts and multi-Lambda deployments.
Recovery runbooks Store runbooks in S3. Lambda fetches the correct runbook for each failure_type, enabling service-specific recovery procedures beyond restart/fallback.
Chaos engineering suite Extend with slow-response, OOM, and partial-failure scenarios using tc (traffic control) or Pumba for realistic fault injection across all registered services.
Dynamic strategy updates Hot-reload services_config.json from S3 on each Lambda invocation so strategy changes take effect without redeploying the Lambda zip.

Why This Project Matters

Modern distributed systems fail. Databases go down, pods crash, deployments introduce regressions. The standard response is human: an on-call engineer wakes up at 3 AM, reads logs, and runs kubectl rollout restart. That loop takes minutes.

This project compresses that loop to seconds — and makes the decisions smarter.

The architecture mirrors patterns used in production at large-scale engineering organizations:

  • Netflix Hystrix / Resilience4j — circuit breakers protecting service meshes from cascading failures
  • AWS Auto Scaling — health-check-driven replacement of unhealthy instances
  • Kubernetes liveness probes + restart policies — container-level automatic recovery
  • PagerDuty + runbook automation — event-driven remediation without human toil
  • Progressive severity escalation — the same pattern used by major incident management platforms (PagerDuty, OpsGenie) to route alerts to the right responder at the right urgency level
  • Service mesh policy (Istio/Linkerd) — per-service traffic management policies, mirrored here as per-service recovery strategies in the registry

The difference is that this system is fully observable, fully tested, and demonstrates these patterns end-to-end in a real AWS environment. Every component — the circuit breaker state machine, the event cooldown, the severity-based decision engine, the strategy-aware service registry, the token-authenticated recovery agent, the 20-widget CloudWatch dashboard — is a simplified but structurally faithful implementation of what runs in real production infrastructure.

For an SRE or DevOps engineer, this is a working demonstration of toil reduction: turning reactive firefighting into proactive, automated, and escalation-aware recovery across an entire service fleet.

Author

Prashant Agrawal

GitHub

License

This project is licensed under the MIT License — see the LICENSE file for details.

Star this repo if it helped you understand self-healing infrastructure patterns.

Built to demonstrate production-grade SRE patterns: circuit breaking, intelligent escalation, strategy-aware recovery, and multi-service observability.

About

Self-healing microservices on AWS - detects failures and auto-recovers via EventBridge, Lambda, and a recovery agent. Per-service strategies (restart/escalate/fallback), circuit breaker, severity escalation, service registry, and 20-widget CloudWatch observability.

54.224.134.71

Topics

aws-lambdacircuit-breakercloudwatchdevopsdistributed-systemsdockerec2eventbridgefastapifault-tolerancemicroservicesobservabilitypythonself-healingsre

Resources

Readme
MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages