Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/operations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ It is intended for developers, operators, and support engineers who need repeata
- `monitor-operations-runbook.md` - day-to-day operational procedures, smoke checks, and incident-oriented troubleshooting
- `kafka-sasl-ssl-quickstart.md` - secure Kafka enablement (TLS/SASL) with bootstrap scripts
- `kafka-sasl-ssl-troubleshooting.md` - failure patterns and diagnostics for Kafka secure mode
- `kafka-sasl-ssl-acceptance-checklist.md` - release checklist to close secure Kafka rollout

### Scope

Expand Down Expand Up @@ -56,6 +57,7 @@ Esta pensada para desarrolladores, operadores y personal de soporte que necesite
- `monitor-operations-runbook.md` - procedimientos operativos diarios, smoke checks y troubleshooting orientado a incidentes
- `kafka-sasl-ssl-quickstart.md` - habilitacion de Kafka seguro (TLS/SASL) con scripts de bootstrap
- `kafka-sasl-ssl-troubleshooting.md` - guia de diagnostico para fallas de Kafka en modo seguro
- `kafka-sasl-ssl-acceptance-checklist.md` - checklist de aceptacion para cierre operativo de Kafka seguro

### Alcance

Expand Down
34 changes: 34 additions & 0 deletions docs/operations/kafka-sasl-ssl-acceptance-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Kafka SASL/SSL Acceptance Checklist

Use this checklist to declare Hallazgo #12 operationally closed.

## Preconditions
- [ ] `.env` contains secure Kafka credentials
- [ ] TLS artifacts exist under `docker/kafka/secrets/`
- [ ] Preflight passes: `./scripts/kafka-secure-preflight.sh`
Comment on lines +6 to +8

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The checklist says ".env contains secure Kafka credentials", but the referenced preflight script (./scripts/kafka-secure-preflight.sh) only checks the current process environment and does not source .env. This can confuse operators who rely on .env alone. Consider updating the checklist to explicitly say the vars must be exported in the shell (or that .env must be sourced before running preflight/smoke scripts).

Suggested change
- [ ] `.env` contains secure Kafka credentials
- [ ] TLS artifacts exist under `docker/kafka/secrets/`
- [ ] Preflight passes: `./scripts/kafka-secure-preflight.sh`
- [ ] Secure Kafka credentials are available as environment variables (exported in the shell or by running `source .env`)
- [ ] TLS artifacts exist under `docker/kafka/secrets/`
- [ ] Preflight passes after env is prepared: `./scripts/kafka-secure-preflight.sh`

Copilot uses AI. Check for mistakes.

## Activation
- [ ] Secure mode applied: `./scripts/kafka-enable-secure-mode.sh`
- [ ] Overlay is active: `docker compose -f docker-compose.yml -f docker-compose.secure.yml ps`

## Smoke validation
- [ ] `./scripts/kafka-secure-smoke.sh` exits with code 0
- [ ] `kafka`, `kafka-connect`, `backend` are running
- [ ] Backend env has `KAFKA_SECURITY_PROTOCOL=SASL_SSL`
- [ ] Connect env has `CONNECT_SECURITY_PROTOCOL=SASL_SSL`
- [ ] Recent logs have no TLS/SASL handshake/auth failures

## Functional validation
- [ ] Debezium connector stays healthy
- [ ] CDC events still arrive to backend pipeline
- [ ] No regression on SSE event delivery

## Rollback validated
- [ ] Plaintext fallback works: `docker compose -f docker-compose.yml up -d`
- [ ] Services recover after rollback

## Evidence to attach in PR/issue
- [ ] `docker compose ps` output (secure mode)
- [ ] `./scripts/kafka-secure-preflight.sh` output
- [ ] `./scripts/kafka-secure-smoke.sh` output
- [ ] relevant logs for kafka/connect/backend
6 changes: 6 additions & 0 deletions docs/operations/kafka-sasl-ssl-quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,12 @@ If preflight fails, fix missing env vars/files and rerun.
- Backend starts with `KAFKA_SECURITY_PROTOCOL=SASL_SSL`
- Debezium connect worker starts with SASL settings

Run automated smoke check:

```bash
./scripts/kafka-secure-smoke.sh
```

## 5) Rollback to plaintext

```bash
Expand Down
72 changes: 72 additions & 0 deletions scripts/kafka-secure-smoke.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
#!/usr/bin/env bash
set -euo pipefail

ROOT_DIR="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT_DIR"

services=(kafka kafka-connect backend)
failed=0

echo "[1/4] Checking container status"
for svc in "${services[@]}"; do
if ! docker compose ps "$svc" >/dev/null 2>&1; then
echo " [FAIL] service not found: $svc"
failed=1
continue
fi

status="$(docker compose ps --format json "$svc" | tr -d '\n' | sed 's/}\s*{/}\n{/g' | grep -m1 '"Service":"'"$svc"'"' || true)"
if [[ -z "$status" ]]; then
echo " [FAIL] unable to read status: $svc"
failed=1
continue
fi

if echo "$status" | grep -q '"State":"running"'; then
Comment on lines +18 to +25

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The container-state detection is parsing docker compose ps --format json output with sed 's/}\s*{/}... where \s is not portable in POSIX/GNU sed (it won’t match whitespace). More importantly, grepping JSON is brittle and can misreport when a service has multiple containers (any "State":"running" match passes). Prefer using the supported Go-template output (e.g., --format '{{.State}}') and explicitly fail if any returned state is not running.

Suggested change
status="$(docker compose ps --format json "$svc" | tr -d '\n' | sed 's/}\s*{/}\n{/g' | grep -m1 '"Service":"'"$svc"'"' || true)"
if [[ -z "$status" ]]; then
echo " [FAIL] unable to read status: $svc"
failed=1
continue
fi
if echo "$status" | grep -q '"State":"running"'; then
states="$(docker compose ps --format '{{.State}}' "$svc" || true)"
if [[ -z "$states" ]]; then
echo " [FAIL] unable to read status: $svc"
failed=1
continue
fi
all_running=1
while IFS= read -r state; do
if [[ "$state" != "running" ]]; then
all_running=0
fi
done <<< "$states"
if [[ "$all_running" -eq 1 ]]; then

Copilot uses AI. Check for mistakes.
echo " [OK] $svc running"
else
echo " [FAIL] $svc not running"
failed=1
fi
done

echo "[2/4] Verifying secure env in backend"
backend_env="$(docker compose exec -T backend /bin/sh -lc 'env' 2>/dev/null || true)"
if echo "$backend_env" | grep -q '^KAFKA_SECURITY_PROTOCOL=SASL_SSL$'; then
echo " [OK] backend KAFKA_SECURITY_PROTOCOL=SASL_SSL"
else
echo " [FAIL] backend missing KAFKA_SECURITY_PROTOCOL=SASL_SSL"
failed=1
fi

echo "[3/4] Verifying secure env in kafka-connect"
connect_env="$(docker compose exec -T kafka-connect /bin/sh -lc 'env' 2>/dev/null || true)"
if echo "$connect_env" | grep -q '^CONNECT_SECURITY_PROTOCOL=SASL_SSL$'; then
echo " [OK] connect CONNECT_SECURITY_PROTOCOL=SASL_SSL"
else
echo " [FAIL] connect missing CONNECT_SECURITY_PROTOCOL=SASL_SSL"
failed=1
fi

echo "[4/4] Scanning logs for obvious auth/ssl failures"
kafka_logs="$(docker compose logs --tail=200 kafka 2>/dev/null || true)"
connect_logs="$(docker compose logs --tail=200 kafka-connect 2>/dev/null || true)"
backend_logs="$(docker compose logs --tail=200 backend 2>/dev/null || true)"
combined_logs="$kafka_logs
$connect_logs
$backend_logs"

if echo "$combined_logs" | grep -Eiq 'SSLHandshakeException|SaslAuthenticationException|PKIX path building failed|No route to host|Connection refused'; then
echo " [WARN] suspicious errors detected in recent logs"
echo " inspect with: docker compose logs --tail=200 kafka kafka-connect backend"
failed=1

Copilot AI Apr 2, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This block prints [WARN] but still sets failed=1, which makes the script exit non-zero. Either treat these matches as [FAIL] (and keep failing), or keep [WARN] but don’t flip failed so operators can distinguish a warning-only run from a hard failure.

Suggested change
failed=1

Copilot uses AI. Check for mistakes.
else
echo " [OK] no obvious TLS/SASL errors in recent logs"
fi

if [[ "$failed" -ne 0 ]]; then
echo "Secure smoke test FAILED"
exit 1
fi

echo "Secure smoke test PASSED"
Loading