fix(security): stabilize kafka secure bootstrap readiness by careb36 · Pull Request #45 · careb36/Monitor

careb36 · 2026-04-02T19:05:19Z

Summary

add broker SCRAM env vars and bootstrap wiring for secure Kafka startup
fix cp-kafka zk preflight behavior under broker JAAS by explicitly setting ZOOKEEPER_SASL_ENABLED=false
generate broker certs with SANs (kafka, localhost, monitor-kafka) to prevent inter-broker/controller TLS hostname failures
extend secure preflight to require broker JAAS/admin client artifacts

Validation

ran ./scripts/kafka-enable-secure-mode.sh end-to-end
secure stack reached healthy state (kafka healthy, backend/kafka-connect started)
ran ./scripts/kafka-secure-smoke.sh => Secure smoke test PASSED

Scope

.env.example
docker-compose.secure.yml
scripts/kafka-generate-secrets.sh
scripts/kafka-bootstrap-scram-users.sh
scripts/kafka-secure-preflight.sh

kilo-code-bot · 2026-04-02T19:06:19Z

 KAFKA_SECURITY_PROTOCOL=PLAINTEXT
 KAFKA_SASL_MECHANISM=SCRAM-SHA-512
+KAFKA_BROKER_USERNAME=broker-user
+KAFKA_BROKER_PASSWORD=


WARNING: Empty password default - ensure users set a secure password for KAFKA_BROKER_PASSWORD before running in secure mode.

kilo-code-bot · 2026-04-02T19:06:34Z

      KAFKA_INTER_BROKER_LISTENER_NAME: SASL_SSL
      KAFKA_SASL_ENABLED_MECHANISMS: SCRAM-SHA-512
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: SCRAM-SHA-512
+      ZOOKEEPER_SASL_ENABLED: "false"


WARNING: Empty password default for KAFKA_BROKER_PASSWORD in .env.example - ensure users set a secure password before running in secure mode.

kilo-code-bot · 2026-04-02T19:06:37Z

 SECRETS_DIR="$ROOT_DIR/docker/kafka/secrets"

 required_env=(
+  KAFKA_BROKER_USERNAME


SUGGESTION: Consider adding a check to ensure password variables contain non-empty values. Currently, setting KAFKA_BROKER_PASSWORD="" would pass preflight but fail at runtime.

kilo-code-bot · 2026-04-02T19:06:43Z

Code Review Summary

Status: 2 Issues Found | Recommendation: Address before merge

Overview

Severity	Count
CRITICAL	0
WARNING	2
SUGGESTION	0

Issue Details (click to expand)

WARNING

File	Line	Issue
`.env.example`	13	Empty password default - ensure users understand they must set a secure password for `KAFKA_BROKER_PASSWORD` before running in secure mode.
`scripts/kafka-secure-preflight.sh`	8	Only checks if variables are set (`-z`), not if they contain non-empty values. A user could set `KAFKA_BROKER_PASSWORD=""` and pass preflight but fail at runtime. Consider adding validation for non-empty values.

Files Reviewed (5 files)

.env.example - 1 issue
docker-compose.secure.yml - 0 issues
scripts/kafka-generate-secrets.sh - 0 issues
scripts/kafka-bootstrap-scram-users.sh - 0 issues
scripts/kafka-secure-preflight.sh - 1 issue

The implementation adds broker SCRAM authentication for secure Kafka mode. The changes are well-structured and follow existing patterns. The main concerns are around ensuring users provide non-empty passwords before running in secure mode.

_{Reviewed by minimax-m2.5-20260211 · 447,245 tokens}

Copilot

Pull request overview

Stabilizes the Docker-based “Kafka secure mode” bootstrap by adding broker SCRAM credentials/artifacts to preflight, wiring broker JAAS into the secure compose overlay, and improving TLS certificate SAN coverage to prevent hostname-related TLS failures.

Changes:

Add broker SCRAM env vars and require new broker JAAS/admin-client artifacts during secure preflight.
Generate broker certs with SANs and emit kafka_server_jaas.conf + kafka_admin_client.properties during secret generation.
Update secure compose overlay to disable ZooKeeper SASL under broker JAAS, add a Kafka healthcheck, and wire endpoint-identification env vars.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`scripts/kafka-secure-preflight.sh`	Extends secure-mode preflight to require broker SCRAM env vars and new broker/admin artifacts.
`scripts/kafka-generate-secrets.sh`	Adds SAN-enabled cert generation and emits broker JAAS + admin client properties.
`scripts/kafka-bootstrap-scram-users.sh`	Bootstraps broker SCRAM user alongside backend/connect users.
`docker-compose.secure.yml`	Wires broker JAAS, disables ZooKeeper SASL, adds Kafka healthcheck, and sets Kafka/Connect TLS envs.
`.env.example`	Documents broker SCRAM env vars for secure-mode runs.

Copilot · 2026-04-02T19:08:58Z

+      CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
      CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_PRODUCER_SASL_MECHANISM: SCRAM-SHA-512
      CONNECT_PRODUCER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";
      CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks
      CONNECT_PRODUCER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}
+      CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
      CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL
      CONNECT_CONSUMER_SASL_MECHANISM: SCRAM-SHA-512
      CONNECT_CONSUMER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";
      CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks
      CONNECT_CONSUMER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}
+      CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:


CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM is set to an empty value, which disables TLS hostname verification for Kafka Connect. Since the broker certs now include SANs for kafka/localhost, this should remain enabled (remove this override or set it to HTTPS) to avoid weakening transport security.

Suggested change

CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:

CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL

CONNECT_PRODUCER_SASL_MECHANISM: SCRAM-SHA-512

CONNECT_PRODUCER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";

CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks

CONNECT_PRODUCER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}

CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:

CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL

CONNECT_CONSUMER_SASL_MECHANISM: SCRAM-SHA-512

CONNECT_CONSUMER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";

CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks

CONNECT_CONSUMER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}

CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:

CONNECT_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

CONNECT_PRODUCER_SECURITY_PROTOCOL: SASL_SSL

CONNECT_PRODUCER_SASL_MECHANISM: SCRAM-SHA-512

CONNECT_PRODUCER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";

CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks

CONNECT_PRODUCER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}

CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

CONNECT_CONSUMER_SECURITY_PROTOCOL: SASL_SSL

CONNECT_CONSUMER_SASL_MECHANISM: SCRAM-SHA-512

CONNECT_CONSUMER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";

CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks

CONNECT_CONSUMER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}

CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

Copilot · 2026-04-02T19:08:59Z

      CONNECT_PRODUCER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";
      CONNECT_PRODUCER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks
      CONNECT_PRODUCER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}
+      CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:


CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM is set to an empty value, disabling hostname verification for producer connections. With SAN-correct certs, prefer leaving the default (HTTPS) rather than disabling verification.

Suggested change

CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:

CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

Copilot · 2026-04-02T19:08:59Z

      CONNECT_CONSUMER_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_CONNECT_USERNAME}" password="${KAFKA_CONNECT_PASSWORD}";
      CONNECT_CONSUMER_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/connect.truststore.jks
      CONNECT_CONSUMER_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_CONNECT_TRUSTSTORE_PASSWORD}
+      CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
    volumes:


CONNECT_CONSUMER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM is set to an empty value, disabling hostname verification for consumer connections. Please remove this or set it to HTTPS so TLS verifies the broker hostname.

Copilot · 2026-04-02T19:08:59Z

      KAFKA_SASL_MECHANISM: SCRAM-SHA-512
      KAFKA_SASL_JAAS_CONFIG: org.apache.kafka.common.security.scram.ScramLoginModule required username="${KAFKA_BACKEND_USERNAME}" password="${KAFKA_BACKEND_PASSWORD}";
      KAFKA_SSL_TRUSTSTORE_LOCATION: /etc/kafka/secrets/backend.truststore.jks
      KAFKA_SSL_TRUSTSTORE_PASSWORD: ${KAFKA_BACKEND_TRUSTSTORE_PASSWORD}
+      KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
    volumes:


KAFKA_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM is set to an empty value, which disables TLS hostname verification for the backend Kafka client. Given the new SAN-enabled broker certs, keep hostname verification enabled (omit this variable or set it to HTTPS).

Copilot · 2026-04-02T19:08:59Z

+sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="${BROKER_USER}" password="${BROKER_PASS}";
+ssl.truststore.location=/etc/kafka/secrets/kafka.truststore.jks
+ssl.truststore.password=${TRUSTPASS}
+ssl.endpoint.identification.algorithm=


The generated kafka_admin_client.properties explicitly sets ssl.endpoint.identification.algorithm= (blank), which disables TLS hostname verification for the admin client used in healthchecks. With SANs being generated now, this should be enabled (remove the line or set it to HTTPS) to avoid masking hostname/cert issues.

Suggested change

ssl.endpoint.identification.algorithm=

Copilot · 2026-04-02T19:09:00Z

+BROKER_USER="${KAFKA_BROKER_USERNAME:-broker-user}"
+BROKER_PASS="${KAFKA_BROKER_PASSWORD:-broker-password}"


BROKER_PASS (and BROKER_USER) fall back to hard-coded defaults, but the secure preflight now requires these env vars to be explicitly set. Keeping defaults here can silently generate JAAS/admin artifacts with one credential while the user later sets different values, leading to confusing auth failures. Consider failing fast if these env vars are missing/empty (or have preflight allow defaults), so all steps stay consistent.

Suggested change

BROKER_USER="${KAFKA_BROKER_USERNAME:-broker-user}"

BROKER_PASS="${KAFKA_BROKER_PASSWORD:-broker-password}"

BROKER_USER="${KAFKA_BROKER_USERNAME:?KAFKA_BROKER_USERNAME must be set and non-empty}"

BROKER_PASS="${KAFKA_BROKER_PASSWORD:?KAFKA_BROKER_PASSWORD must be set and non-empty}"

Copilot · 2026-04-02T19:09:00Z

+BROKER_USER="${KAFKA_BROKER_USERNAME:-broker-user}"
+BROKER_PASS="${KAFKA_BROKER_PASSWORD:-broker-password}"


BROKER_USER / BROKER_PASS default values mean this script can create SCRAM credentials that don’t match the values later required by kafka-secure-preflight.sh, causing hard-to-debug startup failures. Align behavior by requiring these vars to be set (fail fast) rather than defaulting to broker-user/broker-password.

Suggested change

BROKER_USER="${KAFKA_BROKER_USERNAME:-broker-user}"

BROKER_PASS="${KAFKA_BROKER_PASSWORD:-broker-password}"

BROKER_USER="${KAFKA_BROKER_USERNAME:?KAFKA_BROKER_USERNAME must be set}"

BROKER_PASS="${KAFKA_BROKER_PASSWORD:?KAFKA_BROKER_PASSWORD must be set}"

fix(security): stabilize kafka sasl_ssl bootstrap and tls readiness

2d52f09

Copilot AI review requested due to automatic review settings April 2, 2026 19:05

careb36 merged commit 8edb7ac into develop Apr 2, 2026
7 checks passed

careb36 deleted the feature/security-hardening-phase9 branch April 2, 2026 19:05

Copilot started reviewing on behalf of careb36 April 2, 2026 19:05 View session

careb36 mentioned this pull request Apr 2, 2026

chore(release): promote develop to main after security hardening #44

Merged

kilo-code-bot Bot reviewed Apr 2, 2026

View reviewed changes

Copilot AI reviewed Apr 2, 2026

View reviewed changes

	CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:
	CONNECT_PRODUCER_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM: HTTPS

		BROKER_USER="${KAFKA_BROKER_USERNAME:-broker-user}"
		BROKER_PASS="${KAFKA_BROKER_PASSWORD:-broker-password}"

Conversation

careb36 commented Apr 2, 2026

Summary

Validation

Scope

Uh oh!

Uh oh!

kilo-code-bot Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

kilo-code-bot Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

kilo-code-bot Bot Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

kilo-code-bot Bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review Summary

Overview

WARNING

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kilo-code-bot Bot commented Apr 2, 2026 •

edited

Loading