Runtime Configuration Auditing and Drift Detection

[JamesEjembi]

Problem Statement

Production configuration drifts from expected values during incident response or manual fixes, creating silent outages that are hard to diagnose. Implement runtime config auditing that snapshots current config at regular intervals, compares to a committed baseline, and alerts on drift.

Technical Bounds

• Snapshot interval: 5 minutes for all services.
• Baseline: committed config files in repository (config/*.baseline.json).
• Drift types: value change, added key, removed key, type change.
• Alert severity: critical for security-related keys (TLS, auth), warning for performance keys.
• Alert channel: PagerDuty for critical, Slack for warning.
• Dashboard: Grafana panel showing drift history with per-key timeline.

Steps

1. Implement config snapshot collector: GET /config/snapshot on each service, returns full config.
2. Build diff comparator: compare snapshot against baseline, categorize drifts by type and severity.
3. Store drift events in PostgreSQL config_drift_events table.
4. Implement alert routing: PagerDuty API for critical drifts, Slack webhook for warnings.
5. Build Grafana dashboard: per-service drift count over time, latest drifts table.
6. Add auto-remediation: for known safe drifts (auto-scaled values), update baseline automatically with annotation.