Docker Swarm Monitoring Stack

An all-in-one monitoring stack for Docker Swarm environments.

Stack Components

• Docker Swarm
• Elastic Stack (Elasticsearch, Logstash, Kibana, Filebeat, Fleet Server, and Elastic Agent)
• Kafka (KRaft mode — no Zookeeper required)
• Prometheus & Grafana
• Node Exporter & cAdvisor
• Nginx
• ElastAlert

Features

• "Elastic"-based log monitoring pipeline using the Elastic Stack and Kafka

  • Fully supports global Elasticsearch cluster deployment without hardcoding
  • Full Elasticsearch security integration
  • Integration with Elastic Agent and Fleet
  • Example Logstash configuration included
  • Nginx proxy for Elasticsearch
  • Real-time alerting via ElastAlert

• Docker Swarm node and service monitoring using Prometheus and Grafana

  • Collects Swarm metrics using Node Exporter and cAdvisor
  • Includes two pre-built dashboards

• Shell script for easy monitoring stack control

Setup

Requirements: A Linux environment with Docker, Docker Compose, and an active Docker Swarm cluster. If you're unsure how to set this up, refer to BASIC_GUIDE.md.

1. Increase vm.max_map_count for Elasticsearch

vi /etc/sysctl.conf

# Add or update the following line
vm.max_map_count=262144

# Apply the changes
sysctl -p

2. Configure daemon.json to expose Docker metrics for monitoring containers

vi /etc/docker/daemon.json

# Example configuration
{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

3. Run Filebeat on the server to be monitored

See filebeat/README.md for details.

4. Start the Kafka server

Refer to kafka/README.md.

5. Configure Logstash to receive input from Kafka

Modify the bootstrap_servers setting in the Logstash pipeline to point to your Kafka server's IP.

# logstash/pipeline/logstash.conf

input {
  kafka {
    bootstrap_servers => "your_kafka_server_ip:9094"
    topics => ["test_topic"]
    codec => json
    decorate_events => true
  }
}

6. Set up Elasticsearch security

To use all features, basic security must be enabled in Elasticsearch. Follow the minimal security and basic security guides. Copy the generated certificate to ./elasticsearch/config.

If you do not need security, you can disable it in elasticsearch/config/elasticsearch.yml.

7. Set up Fleet Server security

After enabling basic security, you can create certificates for Fleet Server. Follow the Fleet server setup guide and certificate generation guide. Copy the certificates to ./fleet-server.

Fleet Server is optional. If not needed, you can remove the related configuration.

8. Configure environment variables

Copy .env.example to .env and adjust the stack version and credentials. monitor_service.sh loads it automatically, and .env is git-ignored so secrets stay out of version control.

cp .env.example .env
vi .env

Running the Monitoring Stack

Use the provided shell script to start, stop, or manage the monitoring stack:

sh monitor_service.sh start
sh monitor_service.sh stop
sh monitor_service.sh restart
sh monitor_service.sh status
sh monitor_service.sh logs {service_name}
sh monitor_service.sh update {service_name}

Access Dashboards

Once all services are running, you can access the following dashboards:

Service	URL	Description
Grafana	http://<your_server_ip>:3000	Credentials from .env (GRAFANA_ADMIN_USER / GRAFANA_ADMIN_PASSWORD)
Kibana	http://<your_server_ip>:5601

Component Versions

Image versions are pinned (as of July 2026):

Component	Version
Elastic Stack (Elasticsearch, Logstash, Kibana, Elastic Agent)	9.4.3 (set via ELASTIC_VERSION in .env)
Kafka	apache/kafka 4.1.2 (KRaft)
Prometheus	v3.13.0
Grafana	13.1.0
Node Exporter	v1.11.1
cAdvisor	v0.60.3 (ghcr.io/google/cadvisor)
ElastAlert 2	2.30.0

Upgrading an existing 8.x Elasticsearch cluster? Set ELASTIC_VERSION to the latest 8.19.x release first and follow the official upgrade path to 9.x.