diff --git a/docs/guides/druid/rotate-auth/guide.md b/docs/guides/druid/rotate-auth/guide.md index 9955a5bbd0..0bc5b7f062 100644 --- a/docs/guides/druid/rotate-auth/guide.md +++ b/docs/guides/druid/rotate-auth/guide.md @@ -374,7 +374,7 @@ spec: authentication: secretRef: kind: Secret - name: sample-druid-auth-user + name: druid-quickstart-auth-user timeout: 5m apply: IfReady ``` @@ -417,7 +417,7 @@ Spec: Apply: IfReady Authentication: Secret Ref: - Name: sample-druid-auth-user + Name: druid-quickstart-auth-user Database Ref: Name: druid-quickstart Timeout: 5m diff --git a/docs/guides/elasticsearch/gitops/gitops.md b/docs/guides/elasticsearch/gitops/gitops.md index dd9ecc6153..e29d930b1c 100644 --- a/docs/guides/elasticsearch/gitops/gitops.md +++ b/docs/guides/elasticsearch/gitops/gitops.md @@ -345,15 +345,13 @@ At first, we will create a secret containing `user.conf` file with required conf To know more about this configuration file, check [here](/docs/guides/elasticsearch/configuration/combined-cluster/index.md) ```yaml apiVersion: v1 -stringData: - user.conf: | - max_connections=200 - shared_buffers=256MB kind: Secret metadata: name: es-configuration namespace: demo -type: Opaque +stringData: + elasticsearch.yml: |- + indices.query.bool.max_clause_count: 2048 ``` Now, we will add this file to `kubedb/es-configuration.yaml`. diff --git a/docs/guides/elasticsearch/reconfigure/_index.md b/docs/guides/elasticsearch/reconfigure/_index.md new file mode 100644 index 0000000000..cc0d807429 --- /dev/null +++ b/docs/guides/elasticsearch/reconfigure/_index.md @@ -0,0 +1,10 @@ +--- +title: Reconfigure +menu: + docs_{{ .version }}: + identifier: es-reconfigure + name: Reconfigure + parent: es-elasticsearch-guides + weight: 55 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/elasticsearch/reconfigure/elasticsearch-combined.md b/docs/guides/elasticsearch/reconfigure/elasticsearch-combined.md new file mode 100644 index 0000000000..9db90f8833 --- /dev/null +++ b/docs/guides/elasticsearch/reconfigure/elasticsearch-combined.md @@ -0,0 +1,398 @@ +--- +title: Reconfigure Elasticsearch Combined Cluster +menu: + docs_{{ .version }}: + identifier: es-reconfigure-combined + name: Combined Cluster + parent: es-reconfigure + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfigure Elasticsearch Combined Cluster + +This guide will show you how to use `KubeDB` Ops-manager operator to reconfigure an Elasticsearch Combined cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Elasticsearch](/docs/guides/elasticsearch/concepts/elasticsearch/index.md) + - [Combined Cluster](/docs/guides/elasticsearch/clustering/combined-cluster/index.md) + - [ElasticsearchOpsRequest](/docs/guides/elasticsearch/concepts/elasticsearch-ops-request/index.md) + - [Reconfigure Overview](/docs/guides/elasticsearch/reconfigure/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/elasticsearch](/docs/examples/elasticsearch) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +Now, we are going to deploy an `Elasticsearch` Combined cluster using a supported version by `KubeDB` operator. Then we are going to apply `ElasticsearchOpsRequest` to reconfigure its configuration. + +### Prepare Elasticsearch Combined Cluster + +Now, we are going to deploy an `Elasticsearch` combined cluster with version `xpack-8.19.9`. + +### Deploy Elasticsearch + +At first, we will create a secret with the `elasticsearch.yml` file containing required configuration settings. + +**elasticsearch.yml:** + +```yaml +indices.query.bool.max_clause_count: 2048 +``` + +Let's create a k8s secret containing the above configuration where the file name will be the key and the file-content as the value: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: es-combined-custom-config + namespace: demo +stringData: + elasticsearch.yml: |- + indices.query.bool.max_clause_count: 2048 +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-combined-custom-config.yaml +secret/es-combined-custom-config created +``` + +In this section, we are going to create an Elasticsearch object specifying `spec.configuration.secretName` field to apply this custom configuration. Below is the YAML of the `Elasticsearch` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1 +kind: Elasticsearch +metadata: + name: es-combined + namespace: demo +spec: + version: xpack-8.19.9 + enableSSL: true + replicas: 2 + configuration: + secretName: es-combined-custom-config + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Let's create the `Elasticsearch` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-combined.yaml +elasticsearch.kubedb.com/es-combined created +``` + +Now, wait until `es-combined` has status `Ready`. i.e, + +```bash +$ kubectl get es -n demo +NAME VERSION STATUS AGE +es-combined xpack-8.19.9 Ready 20m +``` + +Now, we will check if the Elasticsearch has started with the custom configuration we have provided. + +Exec into the Elasticsearch pod and query the cluster settings to see the configuration: + +```bash +$ kubectl exec -it -n demo es-combined-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.indices&pretty" --user "elastic:X4gzeLWqUHKMoQT7" | grep max_clause_count + "max_clause_count" : "2048" + "max_clause_count" : "2048" +``` + +Here, we can see that our given configuration is applied to the Elasticsearch cluster for all nodes. `indices.query.bool.max_clause_count` is set to `2048` from the default value `1024`. + +### Reconfigure using new config secret + +Now we will reconfigure this cluster to set `indices.query.bool.max_clause_count` to `4096`. + +Update our `elasticsearch.yml` file with the new configuration. + +**elasticsearch.yml:** + +```yaml +indices.query.bool.max_clause_count: 4096 +``` + +Then, we will create a new secret with this configuration file. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-es-combined-custom-config + namespace: demo +stringData: + elasticsearch.yml: |- + indices.query.bool.max_clause_count: 4096 +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/new-es-combined-custom-config.yaml +secret/new-es-combined-custom-config created +``` + +#### Create ElasticsearchOpsRequest + +Now, we will use this secret to replace the previous secret using an `ElasticsearchOpsRequest` CR. The `ElasticsearchOpsRequest` yaml is given below, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: ElasticsearchOpsRequest +metadata: + name: esops-reconfigure-combined + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: es-combined + configuration: + configSecret: + name: new-es-combined-custom-config + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are reconfiguring `es-combined` database. +- `spec.type` specifies that we are performing `Reconfigure` on our database. +- `spec.configuration.configSecret.name` specifies the name of the new secret. + +Let's create the `ElasticsearchOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-reconfigure-update-combined.yaml +elasticsearchopsrequest.ops.kubedb.com/esops-reconfigure-combined created +``` + +#### Verify the new configuration is working + +If everything goes well, `KubeDB` Ops-manager operator will update the `configSecret` of `Elasticsearch` object. + +Let's wait for `ElasticsearchOpsRequest` to be `Successful`. Run the following command to watch `ElasticsearchOpsRequest` CR, + +```bash +$ kubectl get elasticsearchopsrequests -n demo +NAME TYPE STATUS AGE +esops-reconfigure-combined Reconfigure Successful 73s +``` + +We can see from the above output that the `ElasticsearchOpsRequest` has succeeded. If we describe the `ElasticsearchOpsRequest` we will get an overview of the steps that were followed to reconfigure the database. + +```bash +$ kubectl describe elasticsearchopsrequest -n demo esops-reconfigure-combined +Name: esops-reconfigure-combined +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: ElasticsearchOpsRequest +Metadata: + Creation Timestamp: 2026-06-15T09:43:13Z + Generation: 1 + Resource Version: 431415 + UID: 25c9da3e-6e1f-4073-a98c-8df9061307e5 +Spec: + Apply: IfReady + Configuration: + Config Secret: + Name: new-es-combined-custom-config + Restart: auto + Database Ref: + Name: es-combined + Max Retries: 1 + Timeout: 5m + Type: Reconfigure +Status: + Conditions: + Last Transition Time: 2026-06-15T09:43:13Z + Message: Elasticsearch ops request is Reconfiguring + Observed Generation: 1 + Reason: Reconfigure + Status: True + Type: Reconfigure + Last Transition Time: 2026-06-15T09:43:18Z + Message: Successfully updated petSets + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2026-06-15T09:43:28Z + Message: pod exists; ConditionStatus:True; PodName:es-combined-0 + Observed Generation: 1 + Status: True + Type: PodExists--es-combined-0 + Last Transition Time: 2026-06-15T09:43:28Z + Message: create es client; ConditionStatus:True; PodName:es-combined-0 + Observed Generation: 1 + Status: True + Type: CreateEsClient--es-combined-0 + Last Transition Time: 2026-06-15T09:43:29Z + Message: evict pod; ConditionStatus:True; PodName:es-combined-0 + Observed Generation: 1 + Status: True + Type: EvictPod--es-combined-0 + Last Transition Time: 2026-06-15T09:43:53Z + Message: create es client; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CreateEsClient + Last Transition Time: 2026-06-15T09:43:43Z + Message: pod exists; ConditionStatus:True; PodName:es-combined-1 + Observed Generation: 1 + Status: True + Type: PodExists--es-combined-1 + Last Transition Time: 2026-06-15T09:43:43Z + Message: create es client; ConditionStatus:True; PodName:es-combined-1 + Observed Generation: 1 + Status: True + Type: CreateEsClient--es-combined-1 + Last Transition Time: 2026-06-15T09:43:44Z + Message: evict pod; ConditionStatus:True; PodName:es-combined-1 + Observed Generation: 1 + Status: True + Type: EvictPod--es-combined-1 + Last Transition Time: 2026-06-15T09:43:58Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2026-06-15T09:43:59Z + Message: Successfully completed the modification process. + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal PauseDatabase 98s KubeDB Ops-manager Operator Pausing Elasticsearch demo/es-combined + Warning pod exists; ConditionStatus:True; PodName:es-combined-0 85s KubeDB Ops-manager Operator pod exists; ConditionStatus:True; PodName:es-combined-0 + Warning create es client; ConditionStatus:True; PodName:es-combined-0 85s KubeDB Ops-manager Operator create es client; ConditionStatus:True; PodName:es-combined-0 + Warning evict pod; ConditionStatus:True; PodName:es-combined-0 84s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:es-combined-0 + Warning create es client; ConditionStatus:False 80s KubeDB Ops-manager Operator create es client; ConditionStatus:False + Warning create es client; ConditionStatus:True 75s KubeDB Ops-manager Operator create es client; ConditionStatus:True + Warning pod exists; ConditionStatus:True; PodName:es-combined-1 70s KubeDB Ops-manager Operator pod exists; ConditionStatus:True; PodName:es-combined-1 + Warning create es client; ConditionStatus:True; PodName:es-combined-1 70s KubeDB Ops-manager Operator create es client; ConditionStatus:True; PodName:es-combined-1 + Warning evict pod; ConditionStatus:True; PodName:es-combined-1 69s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:es-combined-1 + Warning create es client; ConditionStatus:False 65s KubeDB Ops-manager Operator create es client; ConditionStatus:False + Warning create es client; ConditionStatus:True 60s KubeDB Ops-manager Operator create es client; ConditionStatus:True + Normal RestartNodes 55s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Successful 54s KubeDB Ops-manager Operator Successfully reconfigured all elasticsearch nodes. +``` + +Now let's exec into one of the instances and query the cluster settings to check the new configuration. + +```bash +$ kubectl exec -it -n demo es-combined-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.indices&pretty" --user "elastic:X4gzeLWqUHKMoQT7" | grep max_clause_count + "max_clause_count" : "4096" + "max_clause_count" : "4096" +``` + +As we can see from the configuration of the ready Elasticsearch, the value of `indices.query.bool.max_clause_count` has been changed from `2048` to `4096`. So the reconfiguration of the cluster is successful. + + +### Reconfigure using apply config + +Now we will reconfigure this cluster again to set `indices.query.bool.max_clause_count` to `8192`. This time we won't use a new secret. We will use the `applyConfig` field of the `ElasticsearchOpsRequest`. This will merge the new config into the existing secret. + +#### Create ElasticsearchOpsRequest + +Now, we will use the new configuration in the `applyConfig` field in the `ElasticsearchOpsRequest` CR. The `ElasticsearchOpsRequest` yaml is given below, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: ElasticsearchOpsRequest +metadata: + name: esops-reconfigure-apply-combined + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: es-combined + configuration: + applyConfig: + elasticsearch.yml: | + indices.query.bool.max_clause_count: 8192 + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are reconfiguring `es-combined` cluster. +- `spec.type` specifies that we are performing `Reconfigure` on Elasticsearch. +- `spec.configuration.applyConfig` specifies the new configuration that will be merged into the existing secret. + +Let's create the `ElasticsearchOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-reconfigure-apply-combined.yaml +elasticsearchopsrequest.ops.kubedb.com/esops-reconfigure-apply-combined created +``` + +#### Verify the new configuration is working + +If everything goes well, `KubeDB` Ops-manager operator will merge this new config with the existing configuration. + +Let's wait for `ElasticsearchOpsRequest` to be `Successful`. Run the following command to watch `ElasticsearchOpsRequest` CR, + +```bash +$ kubectl get elasticsearchopsrequests -n demo esops-reconfigure-apply-combined +NAME TYPE STATUS AGE +esops-reconfigure-apply-combined Reconfigure Successful 118s +``` + +We can see from the above output that the `ElasticsearchOpsRequest` has succeeded. Now let's exec into one of the instances and check the new configuration. + +```bash +$ kubectl exec -it -n demo es-combined-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.indices&pretty" --user "elastic:X4gzeLWqUHKMoQT7" | grep max_clause_count + "max_clause_count" : "8192" + "max_clause_count" : "8192" +``` + +As we can see from the configuration of the ready Elasticsearch, the value of `indices.query.bool.max_clause_count` has been changed from `4096` to `8192`. So the reconfiguration of the database using the `applyConfig` field is successful. + + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete es -n demo es-combined +kubectl delete elasticsearchopsrequest -n demo esops-reconfigure-apply-combined esops-reconfigure-combined +kubectl delete secret -n demo es-combined-custom-config new-es-combined-custom-config +kubectl delete namespace demo +``` + +## Next Steps + +- Detail concepts of [Elasticsearch object](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). +- Different Elasticsearch topology clustering modes [here](/docs/guides/elasticsearch/clustering/combined-cluster/index.md). +- Monitor your Elasticsearch database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/elasticsearch/monitoring/using-prometheus-operator.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/elasticsearch/reconfigure/elasticsearch-topology.md b/docs/guides/elasticsearch/reconfigure/elasticsearch-topology.md new file mode 100644 index 0000000000..8628e5951d --- /dev/null +++ b/docs/guides/elasticsearch/reconfigure/elasticsearch-topology.md @@ -0,0 +1,521 @@ +--- +title: Reconfigure Elasticsearch Topology Cluster +menu: + docs_{{ .version }}: + identifier: es-reconfigure-topology + name: Topology Cluster + parent: es-reconfigure + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfigure Elasticsearch Topology Cluster + +This guide will show you how to use `KubeDB` Ops-manager operator to reconfigure an Elasticsearch Topology cluster. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the `kubectl` command-line tool must be configured to communicate with your cluster. + +- Install `KubeDB` Provisioner and Ops-manager operator in your cluster following the steps [here](/docs/setup/README.md). + +- You should be familiar with the following `KubeDB` concepts: + - [Elasticsearch](/docs/guides/elasticsearch/concepts/elasticsearch/index.md) + - [Topology Cluster](/docs/guides/elasticsearch/clustering/topology-cluster/simple-dedicated-cluster/index.md) + - [ElasticsearchOpsRequest](/docs/guides/elasticsearch/concepts/elasticsearch-ops-request/index.md) + - [Reconfigure Overview](/docs/guides/elasticsearch/reconfigure/overview.md) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +> **Note:** YAML files used in this tutorial are stored in [docs/examples/elasticsearch](/docs/examples/elasticsearch) directory of [kubedb/docs](https://github.com/kubedb/docs) repository. + +Now, we are going to deploy an `Elasticsearch` Topology cluster using a supported version by `KubeDB` operator. Then we are going to apply `ElasticsearchOpsRequest` to reconfigure its configuration. + +### Prepare Elasticsearch Topology Cluster + +Now, we are going to deploy an `Elasticsearch` topology cluster with version `xpack-8.19.9`. + +### Deploy Elasticsearch + +At first, we will create a secret with role-prefixed configuration files. In a topology cluster, you can target specific node roles by prefixing the filename with the role name (e.g., `master-elasticsearch.yml` applies only to master nodes, `data-elasticsearch.yml` applies only to data nodes). + +**master-elasticsearch.yml:** + +```yaml +cluster.max_shards_per_node: 2000 +``` + +**data-elasticsearch.yml:** + +```yaml +indices.query.bool.max_clause_count: 2048 +``` + +**ingest-elasticsearch.yml:** + +```yaml +http.max_content_length: 200mb +``` + +Here, `cluster.max_shards_per_node` is set to `2000` (default `1000`) for master nodes, `indices.query.bool.max_clause_count` is set to `2048` (default `1024`) for data nodes, and `http.max_content_length` is set to `200mb` (default `100mb`) for ingest nodes. + +Let's create a k8s secret containing the above configuration where the file name will be the key and the file-content as the value: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: es-topology-custom-config + namespace: demo +stringData: + master-elasticsearch.yml: |- + cluster.max_shards_per_node: 2000 + data-elasticsearch.yml: |- + indices.query.bool.max_clause_count: 2048 + ingest-elasticsearch.yml: |- + http.max_content_length: 200mb +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-topology-custom-config.yaml +secret/es-topology-custom-config created +``` + +In this section, we are going to create an Elasticsearch object specifying `spec.configuration.secretName` field to apply this custom configuration. Below is the YAML of the `Elasticsearch` CR that we are going to create, + +```yaml +apiVersion: kubedb.com/v1 +kind: Elasticsearch +metadata: + name: es-topology + namespace: demo +spec: + version: xpack-8.19.9 + enableSSL: true + configuration: + secretName: es-topology-custom-config + topology: + master: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + data: + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + ingest: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + storageType: Durable + deletionPolicy: WipeOut +``` + +Let's create the `Elasticsearch` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-topology.yaml +elasticsearch.kubedb.com/es-topology created +``` + +Now, wait until `es-topology` has status `Ready`. i.e, + +```bash +$ kubectl get es -n demo -w +NAME VERSION STATUS AGE +es-topology xpack-8.19.9 Provisioning 0s +es-topology xpack-8.19.9 Provisioning 24s +. +. +es-topology xpack-8.19.9 Ready 92s +``` + +Now, we will check if the Elasticsearch has started with the custom configuration we have provided. + +Exec into the master node and check the cluster setting: + +```bash +$ kubectl exec -it -n demo es-topology-master-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.cluster&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_shards_per_node + "max_shards_per_node" : "2000", +``` + +Exec into a data node and check the index settings: + +```bash +$ kubectl exec -it -n demo es-topology-data-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.indices&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_clause_count + "max_clause_count" : "2048" + "max_clause_count" : "2048" +``` + +Exec into the ingest node and check the HTTP settings: + +```bash +$ kubectl exec -it -n demo es-topology-ingest-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.http&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_content_length + "max_content_length" : "200mb", +``` + +Here, we can see that our given configurations are applied to the respective node roles. + +### Reconfigure using new config secret + +Now we will reconfigure this cluster to update `cluster.max_shards_per_node` to `3000` for master nodes, `indices.query.bool.max_clause_count` to `4096` for data nodes, and `http.max_content_length` to `300mb` for ingest nodes. + +Update our configuration files with the new values. + +**master-elasticsearch.yml:** + +```yaml +cluster.max_shards_per_node: 3000 +``` + +**data-elasticsearch.yml:** + +```yaml +indices.query.bool.max_clause_count: 4096 +``` + +**ingest-elasticsearch.yml:** + +```yaml +http.max_content_length: 300mb +``` + +Then, we will create a new secret with these configuration files. + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: new-es-topology-custom-config + namespace: demo +stringData: + master-elasticsearch.yml: |- + cluster.max_shards_per_node: 3000 + data-elasticsearch.yml: |- + indices.query.bool.max_clause_count: 4096 + ingest-elasticsearch.yml: |- + http.max_content_length: 300mb +``` + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/new-es-topology-custom-config.yaml +secret/new-es-topology-custom-config created +``` + +#### Create ElasticsearchOpsRequest + +Now, we will use this secret to replace the previous secret using an `ElasticsearchOpsRequest` CR. The `ElasticsearchOpsRequest` yaml is given below, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: ElasticsearchOpsRequest +metadata: + name: esops-reconfigure-topology + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: es-topology + configuration: + configSecret: + name: new-es-topology-custom-config + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are reconfiguring `es-topology` database. +- `spec.type` specifies that we are performing `Reconfigure` on our database. +- `spec.configuration.configSecret.name` specifies the name of the new secret. + +Let's create the `ElasticsearchOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-reconfigure-update-topology.yaml +elasticsearchopsrequest.ops.kubedb.com/esops-reconfigure-topology created +``` + +#### Verify the new configuration is working + +If everything goes well, `KubeDB` Ops-manager operator will update the `configSecret` of `Elasticsearch` object. + +Let's wait for `ElasticsearchOpsRequest` to be `Successful`. Run the following command to watch `ElasticsearchOpsRequest` CR, + +```bash +$ kubectl get elasticsearchopsrequests -n demo +NAME TYPE STATUS AGE +esops-reconfigure-topology Reconfigure Successful 4m55s +``` + +We can see from the above output that the `ElasticsearchOpsRequest` has succeeded. If we describe the `ElasticsearchOpsRequest` we will get an overview of the steps that were followed to reconfigure the database. + +```bash +$ kubectl describe elasticsearchopsrequest -n demo esops-reconfigure-topology +Name: esops-reconfigure-topology +Namespace: demo +Labels: +Annotations: +API Version: ops.kubedb.com/v1alpha1 +Kind: ElasticsearchOpsRequest +Metadata: + Creation Timestamp: 2026-06-15T11:13:59Z + Generation: 1 + Resource Version: 434719 + UID: a050a578-420c-4bb8-a390-3b32b369ec65 +Spec: + Apply: IfReady + Configuration: + Config Secret: + Name: new-es-topology-custom-config + Restart: auto + Database Ref: + Name: es-topology + Max Retries: 1 + Timeout: 5m + Type: Reconfigure +Status: + Conditions: + Last Transition Time: 2026-06-15T11:13:59Z + Message: Elasticsearch ops request is Reconfiguring + Observed Generation: 1 + Reason: Reconfigure + Status: True + Type: Reconfigure + Last Transition Time: 2026-06-15T11:14:09Z + Message: Successfully updated petSets + Observed Generation: 1 + Reason: UpdatePetSets + Status: True + Type: UpdatePetSets + Last Transition Time: 2026-06-15T11:14:19Z + Message: pod exists; ConditionStatus:True; PodName:es-topology-ingest-0 + Observed Generation: 1 + Status: True + Type: PodExists--es-topology-ingest-0 + Last Transition Time: 2026-06-15T11:14:19Z + Message: create es client; ConditionStatus:True; PodName:es-topology-ingest-0 + Observed Generation: 1 + Status: True + Type: CreateEsClient--es-topology-ingest-0 + Last Transition Time: 2026-06-15T11:14:19Z + Message: evict pod; ConditionStatus:True; PodName:es-topology-ingest-0 + Observed Generation: 1 + Status: True + Type: EvictPod--es-topology-ingest-0 + Last Transition Time: 2026-06-15T11:14:59Z + Message: create es client; ConditionStatus:True + Observed Generation: 1 + Status: True + Type: CreateEsClient + Last Transition Time: 2026-06-15T11:14:34Z + Message: pod exists; ConditionStatus:True; PodName:es-topology-data-0 + Observed Generation: 1 + Status: True + Type: PodExists--es-topology-data-0 + Last Transition Time: 2026-06-15T11:14:34Z + Message: create es client; ConditionStatus:True; PodName:es-topology-data-0 + Observed Generation: 1 + Status: True + Type: CreateEsClient--es-topology-data-0 + Last Transition Time: 2026-06-15T11:14:34Z + Message: evict pod; ConditionStatus:True; PodName:es-topology-data-0 + Observed Generation: 1 + Status: True + Type: EvictPod--es-topology-data-0 + Last Transition Time: 2026-06-15T11:14:49Z + Message: pod exists; ConditionStatus:True; PodName:es-topology-data-1 + Observed Generation: 1 + Status: True + Type: PodExists--es-topology-data-1 + Last Transition Time: 2026-06-15T11:14:49Z + Message: create es client; ConditionStatus:True; PodName:es-topology-data-1 + Observed Generation: 1 + Status: True + Type: CreateEsClient--es-topology-data-1 + Last Transition Time: 2026-06-15T11:14:49Z + Message: evict pod; ConditionStatus:True; PodName:es-topology-data-1 + Observed Generation: 1 + Status: True + Type: EvictPod--es-topology-data-1 + Last Transition Time: 2026-06-15T11:15:04Z + Message: pod exists; ConditionStatus:True; PodName:es-topology-master-0 + Observed Generation: 1 + Status: True + Type: PodExists--es-topology-master-0 + Last Transition Time: 2026-06-15T11:15:04Z + Message: create es client; ConditionStatus:True; PodName:es-topology-master-0 + Observed Generation: 1 + Status: True + Type: CreateEsClient--es-topology-master-0 + Last Transition Time: 2026-06-15T11:15:04Z + Message: evict pod; ConditionStatus:True; PodName:es-topology-master-0 + Observed Generation: 1 + Status: True + Type: EvictPod--es-topology-master-0 + Last Transition Time: 2026-06-15T11:15:14Z + Message: Successfully restarted all nodes + Observed Generation: 1 + Reason: RestartNodes + Status: True + Type: RestartNodes + Last Transition Time: 2026-06-15T11:15:15Z + Message: Successfully completed the modification process. + Observed Generation: 1 + Reason: Successful + Status: True + Type: Successful + Observed Generation: 1 + Phase: Successful +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal PauseDatabase 4m47s KubeDB Ops-manager Operator Pausing Elasticsearch demo/es-topology + Warning pod exists; ConditionStatus:True; PodName:es-topology-ingest-0 4m29s KubeDB Ops-manager Operator pod exists; ConditionStatus:True; PodName:es-topology-ingest-0 + Warning create es client; ConditionStatus:True; PodName:es-topology-ingest-0 4m29s KubeDB Ops-manager Operator create es client; ConditionStatus:True; PodName:es-topology-ingest-0 + Warning evict pod; ConditionStatus:True; PodName:es-topology-ingest-0 4m29s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:es-topology-ingest-0 + Warning create es client; ConditionStatus:False 4m24s KubeDB Ops-manager Operator create es client; ConditionStatus:False + Warning create es client; ConditionStatus:True 4m19s KubeDB Ops-manager Operator create es client; ConditionStatus:True + Warning pod exists; ConditionStatus:True; PodName:es-topology-data-0 4m14s KubeDB Ops-manager Operator pod exists; ConditionStatus:True; PodName:es-topology-data-0 + Warning create es client; ConditionStatus:True; PodName:es-topology-data-0 4m14s KubeDB Ops-manager Operator create es client; ConditionStatus:True; PodName:es-topology-data-0 + Warning evict pod; ConditionStatus:True; PodName:es-topology-data-0 4m14s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:es-topology-data-0 + Warning create es client; ConditionStatus:False 4m9s KubeDB Ops-manager Operator create es client; ConditionStatus:False + Warning create es client; ConditionStatus:True 4m4s KubeDB Ops-manager Operator create es client; ConditionStatus:True + Warning pod exists; ConditionStatus:True; PodName:es-topology-data-1 3m59s KubeDB Ops-manager Operator pod exists; ConditionStatus:True; PodName:es-topology-data-1 + Warning create es client; ConditionStatus:True; PodName:es-topology-data-1 3m59s KubeDB Ops-manager Operator create es client; ConditionStatus:True; PodName:es-topology-data-1 + Warning evict pod; ConditionStatus:True; PodName:es-topology-data-1 3m59s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:es-topology-data-1 + Warning create es client; ConditionStatus:False 3m54s KubeDB Ops-manager Operator create es client; ConditionStatus:False + Warning create es client; ConditionStatus:True 3m49s KubeDB Ops-manager Operator create es client; ConditionStatus:True + Warning pod exists; ConditionStatus:True; PodName:es-topology-master-0 3m44s KubeDB Ops-manager Operator pod exists; ConditionStatus:True; PodName:es-topology-master-0 + Warning create es client; ConditionStatus:True; PodName:es-topology-master-0 3m44s KubeDB Ops-manager Operator create es client; ConditionStatus:True; PodName:es-topology-master-0 + Warning evict pod; ConditionStatus:True; PodName:es-topology-master-0 3m44s KubeDB Ops-manager Operator evict pod; ConditionStatus:True; PodName:es-topology-master-0 + Warning create es client; ConditionStatus:True 3m39s KubeDB Ops-manager Operator create es client; ConditionStatus:True + Normal RestartNodes 3m34s KubeDB Ops-manager Operator Successfully restarted all nodes + Normal Successful 3m33s KubeDB Ops-manager Operator Successfully reconfigured all elasticsearch nodes. +``` + +Now let's exec into a master node, a data node, and the ingest node to verify the new configuration. + +```bash +$ kubectl exec -it -n demo es-topology-master-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.cluster&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_shards_per_node + "max_shards_per_node" : "3000", + +$ kubectl exec -it -n demo es-topology-data-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.indices&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_clause_count + "max_clause_count" : "4096" + "max_clause_count" : "4096" + +$ kubectl exec -it -n demo es-topology-ingest-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.http&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_content_length + "max_content_length" : "300mb", +``` + +As we can see, the values have been updated on their respective node roles. So the reconfiguration of the cluster is successful. + + +### Reconfigure using apply config + +Now we will reconfigure this cluster again using the `applyConfig` field. This will merge the new config into the existing secret without requiring a new secret. + +#### Create ElasticsearchOpsRequest + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: ElasticsearchOpsRequest +metadata: + name: esops-reconfigure-apply-topology + namespace: demo +spec: + type: Reconfigure + databaseRef: + name: es-topology + configuration: + applyConfig: + master-elasticsearch.yml: | + cluster.max_shards_per_node: 4000 + data-elasticsearch.yml: | + indices.query.bool.max_clause_count: 8192 + ingest-elasticsearch.yml: | + http.max_content_length: 400mb + timeout: 5m + apply: IfReady +``` + +Here, + +- `spec.databaseRef.name` specifies that we are reconfiguring `es-topology` cluster. +- `spec.type` specifies that we are performing `Reconfigure` on Elasticsearch. +- `spec.configuration.applyConfig` specifies the new configuration that will be merged into the existing secret. + +Let's create the `ElasticsearchOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/reconfigure/es-reconfigure-apply-topology.yaml +elasticsearchopsrequest.ops.kubedb.com/esops-reconfigure-apply-topology created +``` + +#### Verify the new configuration is working + +Let's wait for `ElasticsearchOpsRequest` to be `Successful`. + +```bash +$ kubectl get elasticsearchopsrequests -n demo esops-reconfigure-apply-topology +NAME TYPE STATUS AGE +esops-reconfigure-apply-topology Reconfigure Successful 6m52s +``` + +Now let's verify the updated values on each node role. + +```bash +$ kubectl exec -it -n demo es-topology-master-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.cluster&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_shards_per_node + "max_shards_per_node" : "4000", + +$ kubectl exec -it -n demo es-topology-data-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.indices&pretty" --user "elastic:$ELASTIC_USER_PASSWORD" | grep max_clause_count + "max_clause_count" : "8192" + "max_clause_count" : "8192" + +$ kubectl exec -it -n demo es-topology-ingest-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_nodes/settings?filter_path=nodes.*.settings.http&pretty" --user "elastic:X4am_*ihVy~M)m0j" | grep max_content_length + "max_content_length" : "400mb", +``` + +As we can see, `cluster.max_shards_per_node` has been changed from `3000` to `4000` on master nodes, `indices.query.bool.max_clause_count` has been changed from `4096` to `8192` on data nodes, and `http.max_content_length` has been changed from `300mb` to `400mb` on ingest nodes. So the reconfiguration using the `applyConfig` field is successful. + + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete es -n demo es-topology +kubectl delete elasticsearchopsrequest -n demo esops-reconfigure-apply-topology esops-reconfigure-topology +kubectl delete secret -n demo es-topology-custom-config new-es-topology-custom-config +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Elasticsearch object](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). +- Different Elasticsearch topology clustering modes [here](/docs/guides/elasticsearch/clustering/topology-cluster/hot-warm-cold-cluster/index.md). +- Monitor your Elasticsearch database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/elasticsearch/monitoring/using-prometheus-operator.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/elasticsearch/reconfigure/overview.md b/docs/guides/elasticsearch/reconfigure/overview.md new file mode 100644 index 0000000000..012e8f2163 --- /dev/null +++ b/docs/guides/elasticsearch/reconfigure/overview.md @@ -0,0 +1,65 @@ +--- +title: Reconfiguring Elasticsearch +menu: + docs_{{ .version }}: + identifier: es-reconfigure-overview + name: Overview + parent: es-reconfigure + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Reconfiguring Elasticsearch + +This guide will give an overview on how KubeDB Ops-manager operator reconfigures `Elasticsearch` components such as Combined, Topology (master, data, ingest), etc. + +## Before You Begin + +- You should be familiar with the following `KubeDB` concepts: + - [Elasticsearch](/docs/guides/elasticsearch/concepts/elasticsearch/index.md) + - [ElasticsearchOpsRequest](/docs/guides/elasticsearch/concepts/elasticsearch-ops-request/index.md) + +## How Reconfiguring Elasticsearch Process Works + +The following diagram shows how KubeDB Ops-manager operator reconfigures `Elasticsearch` components. Open the image in a new tab to see the enlarged version. + +
+ Reconfiguring process of Elasticsearch +
Fig: Reconfiguring process of Elasticsearch
+
+ +The Reconfiguring Elasticsearch process consists of the following steps: + +1. At first, a user creates an `Elasticsearch` Custom Resource (CR). + +2. `KubeDB` Provisioner operator watches the `Elasticsearch` CR. + +3. When the operator finds an `Elasticsearch` CR, it creates the required number of `PetSets` and related necessary stuff like secrets, services, etc. + +4. Then, in order to reconfigure the various components (ie. Combined, Topology) of the `Elasticsearch`, the user creates an `ElasticsearchOpsRequest` CR with desired information. + +5. `KubeDB` Ops-manager operator watches the `ElasticsearchOpsRequest` CR. + +6. When it finds an `ElasticsearchOpsRequest` CR, it halts the `Elasticsearch` object which is referred from the `ElasticsearchOpsRequest`. So, the `KubeDB` Provisioner operator doesn't perform any operations on the `Elasticsearch` object during the reconfiguring process. + +7. Then the `KubeDB` Ops-manager operator will replace the existing configuration with the new configuration provided or merge the new configuration with the existing configuration according to the `ElasticsearchOpsRequest` CR. + +8. Then the `KubeDB` Ops-manager operator will restart the related PetSet Pods so that they restart with the new configuration defined in the `ElasticsearchOpsRequest` CR. + +9. After the successful reconfiguring of the `Elasticsearch` components, the `KubeDB` Ops-manager operator resumes the `Elasticsearch` object so that the `KubeDB` Provisioner operator resumes its usual operations. +### spec.configuration Fields + +The `ElasticsearchOpsRequest` with `type: Reconfigure` uses the following sub-fields under `spec.configuration`: + +| Field | Description | +|---|---| +| `configSecret` | Reference to a `Secret` containing the full custom configuration file(s) for the database. | +| `secureConfigSecret` | Reference to a `Secret` containing secure settings (e.g., keystore passwords). | +| `applyConfig` | Inline map of `filename: \| content` entries. Merged into the existing `ConfigSecret`. If no `ConfigSecret` exists, a new secret named `{db-name}-user-config` is created. | +| `removeCustomConfig` | Set to `true` to remove all user-provided configuration and revert to the operator-generated defaults. | +| `removeSecureCustomConfig` | Set to `true` to remove user-provided secure settings and revert to the default empty keystore. | + +In the next docs, we are going to show a step by step guide on reconfiguring Elasticsearch components using `ElasticsearchOpsRequest` CRD. diff --git a/docs/guides/elasticsearch/tls/_index.md b/docs/guides/elasticsearch/tls/_index.md new file mode 100644 index 0000000000..d38b896e87 --- /dev/null +++ b/docs/guides/elasticsearch/tls/_index.md @@ -0,0 +1,10 @@ +--- +title: Run Elasticsearch with TLS +menu: + docs_{{ .version }}: + identifier: es-tls + name: TLS/SSL Encryption + parent: es-elasticsearch-guides + weight: 70 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/elasticsearch/tls/elasticsearch-combined.md b/docs/guides/elasticsearch/tls/elasticsearch-combined.md new file mode 100644 index 0000000000..8f15d958b8 --- /dev/null +++ b/docs/guides/elasticsearch/tls/elasticsearch-combined.md @@ -0,0 +1,237 @@ +--- +title: Elasticsearch Combined Cluster TLS/SSL Encryption +menu: + docs_{{ .version }}: + identifier: es-tls-combined + name: Combined Cluster + parent: es-tls + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Run Elasticsearch with TLS/SSL (Transport Encryption) + +KubeDB supports providing TLS/SSL encryption for Elasticsearch. This tutorial will show you how to use KubeDB to run an Elasticsearch combined cluster with TLS/SSL encryption. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install [`cert-manager`](https://cert-manager.io/docs/installation/) v1.0.0 or later to your cluster to manage your SSL/TLS certificates. + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + + ```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/elasticsearch](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/elasticsearch) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Overview + +KubeDB uses the following crd fields to enable SSL/TLS encryption in Elasticsearch. + +- `spec:` + - `enableSSL` + - `tls:` + - `issuerRef` + - `certificates` + +Read about the fields in details in [elasticsearch concept](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). + +`tls` is applicable for all types of Elasticsearch (i.e., `combined` and `topology`). + +Users must specify the `tls.issuerRef` field. KubeDB uses the `issuer` or `clusterIssuer` referenced in the `tls.issuerRef` field, and the certificate specs provided in `tls.certificates` to generate certificate secrets. These certificate secrets are then used to configure TLS for both the transport layer (node-to-node communication) and the HTTP layer (client-to-node communication), containing `ca.crt`, `tls.crt` and `tls.key`. + +> Note: `tls.issuerRef` is optional. A user can deploy Elasticsearch without creating an `Issuer`/`ClusterIssuer` by just setting `enableSSL: true`. In that case, the KubeDB Elasticsearch operator automatically creates a self-signed CA and the necessary certificate secrets. + +## Create Issuer/ ClusterIssuer + +We are going to create an example `Issuer` that will be used throughout the duration of this tutorial to enable SSL/TLS in Elasticsearch. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificate using openssl. + +```bash +openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=elasticsearch/O=kubedb" +``` + +- Now create a ca-secret using the certificate files you have just generated. + +```bash +$ kubectl create secret tls es-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/es-ca created +``` + +Now, create an `Issuer` using the `ca-secret` you have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: es-ca-issuer + namespace: demo +spec: + ca: + secretName: es-ca +``` + +Apply the `YAML` file: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/tls/es-issuer.yaml +issuer.cert-manager.io/es-ca-issuer created +``` + +## TLS/SSL encryption in Elasticsearch Combined Cluster + +```yaml +apiVersion: kubedb.com/v1 +kind: Elasticsearch +metadata: + name: es-combined-tls + namespace: demo +spec: + version: xpack-8.19.9 + enableSSL: true + tls: + issuerRef: + apiGroup: "cert-manager.io" + kind: Issuer + name: es-ca-issuer + replicas: 3 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + storageType: Durable + deletionPolicy: WipeOut +``` + +### Deploy Elasticsearch Combined Cluster + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/tls/es-combined-tls.yaml +elasticsearch.kubedb.com/es-combined-tls created +``` + +Now, wait until `es-combined-tls` has status `Ready`. i.e, + +```bash +$ kubectl get es -n demo -w +NAME VERSION STATUS AGE +es-combined-tls xpack-8.19.9 Provisioning 0s +es-combined-tls xpack-8.19.9 Provisioning 15s +. +. +es-combined-tls xpack-8.19.9 Ready 82s +``` + +### Verify TLS/SSL in Elasticsearch Combined Cluster + +KubeDB creates a client certificate secret for Elasticsearch. Let's check it: + +```bash +$ kubectl describe secret -n demo es-combined-tls-client-cert +Name: es-combined-tls-client-cert +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=es-combined-tls + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=elasticsearches.kubedb.com + controller.cert-manager.io/fao=true +Annotations: cert-manager.io/alt-names: + *.es-combined-tls-pods.demo.svc,*.es-combined-tls-pods.demo.svc.cluster.local,es-combined-tls,es-combined-tls.demo.svc,localhost + cert-manager.io/certificate-name: es-combined-tls-client-cert + cert-manager.io/common-name: es-combined-tls + cert-manager.io/ip-sans: 127.0.0.1 + cert-manager.io/issuer-group: cert-manager.io + cert-manager.io/issuer-kind: Issuer + cert-manager.io/issuer-name: es-ca-issuer + cert-manager.io/subject-organizations: kubedb + cert-manager.io/uri-sans: + +Type: kubernetes.io/tls + +Data +==== +tls.key: 1708 bytes +ca.crt: 1172 bytes +tls.crt: 1387 bytes +``` + +Now, let's exec into an Elasticsearch pod and verify the configuration that TLS is enabled for both transport and HTTP layers. + +```bash +$ kubectl exec -n demo es-combined-tls-0 -c elasticsearch -- \ + cat /usr/share/elasticsearch/config/elasticsearch.yml | grep -A 2 -i xpack.security +xpack.security.enabled: true + +xpack.security.transport.ssl.enabled: true +xpack.security.transport.ssl.verification_mode: certificate +xpack.security.transport.ssl.key: certs/transport/tls.key +xpack.security.transport.ssl.certificate: certs/transport/tls.crt +xpack.security.transport.ssl.certificate_authorities: [ "certs/transport/ca.crt" ] + +xpack.security.http.ssl.enabled: true +xpack.security.http.ssl.key: certs/http/tls.key +xpack.security.http.ssl.certificate: certs/http/tls.crt +xpack.security.http.ssl.certificate_authorities: [ "certs/http/ca.crt" ] +``` + +We can see from the above output that both `xpack.security.transport.ssl.enabled: true` and `xpack.security.http.ssl.enabled: true` are set, which means TLS is enabled for both node-to-node and client-to-node communication. + +Now, let's connect to the Elasticsearch cluster using HTTPS to confirm it is accessible with TLS. + +```bash +$ kubectl exec -it -n demo es-combined-tls-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_cluster/health?pretty" --user "elastic:$ELASTIC_USER_PASSWORD" +{ + "cluster_name" : "es-combined-tls", + "status" : "green", + "timed_out" : false, + "number_of_nodes" : 3, + "number_of_data_nodes" : 3, + "active_primary_shards" : 4, + "active_shards" : 8, + "relocating_shards" : 0, + "initializing_shards" : 0, + "unassigned_shards" : 0, + "unassigned_primary_shards" : 0, + "delayed_unassigned_shards" : 0, + "number_of_pending_tasks" : 0, + "number_of_in_flight_fetch" : 0, + "task_max_waiting_in_queue_millis" : 0, + "active_shards_percent_as_number" : 100.0 +} +``` + +From the above output, we can see that we are able to connect to the Elasticsearch cluster using the TLS configuration. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete es -n demo es-combined-tls +kubectl delete issuer -n demo es-ca-issuer +kubectl delete secret -n demo es-ca +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Elasticsearch object](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). +- Different Elasticsearch topology clustering modes [here](/docs/guides/elasticsearch/clustering/combined-cluster/index.md). +- Monitor your Elasticsearch database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/elasticsearch/monitoring/using-prometheus-operator.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/elasticsearch/tls/elasticsearch-topology.md b/docs/guides/elasticsearch/tls/elasticsearch-topology.md new file mode 100644 index 0000000000..f943feed40 --- /dev/null +++ b/docs/guides/elasticsearch/tls/elasticsearch-topology.md @@ -0,0 +1,256 @@ +--- +title: Elasticsearch Topology Cluster TLS/SSL Encryption +menu: + docs_{{ .version }}: + identifier: es-tls-topology + name: Topology Cluster + parent: es-tls + weight: 30 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Run Elasticsearch with TLS/SSL (Transport Encryption) + +KubeDB supports providing TLS/SSL encryption for Elasticsearch. This tutorial will show you how to use KubeDB to run an Elasticsearch topology cluster with TLS/SSL encryption. + +## Before You Begin + +- At first, you need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. If you do not already have a cluster, you can create one by using [kind](https://kind.sigs.k8s.io/docs/user/quick-start/). + +- Install [`cert-manager`](https://cert-manager.io/docs/installation/) v1.0.0 or later to your cluster to manage your SSL/TLS certificates. + +- Now, install KubeDB cli on your workstation and KubeDB operator in your cluster following the steps [here](/docs/setup/README.md). + +- To keep things isolated, this tutorial uses a separate namespace called `demo` throughout this tutorial. + + ```bash + $ kubectl create ns demo + namespace/demo created + ``` + +> Note: YAML files used in this tutorial are stored in [docs/examples/elasticsearch](https://github.com/kubedb/docs/tree/{{< param "info.version" >}}/docs/examples/elasticsearch) folder in GitHub repository [kubedb/docs](https://github.com/kubedb/docs). + +## Overview + +KubeDB uses the following crd fields to enable SSL/TLS encryption in Elasticsearch. + +- `spec:` + - `enableSSL` + - `tls:` + - `issuerRef` + - `certificates` + +Read about the fields in details in [elasticsearch concept](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). + +`tls` is applicable for all types of Elasticsearch (i.e., `combined` and `topology`). + +Users must specify the `tls.issuerRef` field. KubeDB uses the `issuer` or `clusterIssuer` referenced in the `tls.issuerRef` field, and the certificate specs provided in `tls.certificates` to generate certificate secrets. These certificate secrets are then used to configure TLS for both the transport layer (node-to-node communication) and the HTTP layer (client-to-node communication), containing `ca.crt`, `tls.crt` and `tls.key`. + +> Note: `tls.issuerRef` is optional. A user can deploy Elasticsearch without creating an `Issuer`/`ClusterIssuer` by just setting `enableSSL: true`. In that case, the KubeDB Elasticsearch operator automatically creates a self-signed CA and the necessary certificate secrets. + +## Create Issuer/ ClusterIssuer + +We are going to create an example `Issuer` that will be used throughout the duration of this tutorial to enable SSL/TLS in Elasticsearch. Alternatively, you can follow this [cert-manager tutorial](https://cert-manager.io/docs/configuration/ca/) to create your own `Issuer`. + +- Start off by generating a ca certificate using openssl. + +```bash +openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./ca.key -out ./ca.crt -subj "/CN=elasticsearch/O=kubedb" +``` + +- Now create a ca-secret using the certificate files you have just generated. + +```bash +kubectl create secret tls es-ca \ + --cert=ca.crt \ + --key=ca.key \ + --namespace=demo +secret/es-ca created +``` + +Now, create an `Issuer` using the `ca-secret` you have just created. The `YAML` file looks like this: + +```yaml +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: es-ca-issuer + namespace: demo +spec: + ca: + secretName: es-ca +``` + +Apply the `YAML` file: + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/tls/es-issuer.yaml +issuer.cert-manager.io/es-ca-issuer created +``` + +## TLS/SSL encryption in Elasticsearch Topology Cluster + +```yaml +apiVersion: kubedb.com/v1 +kind: Elasticsearch +metadata: + name: es-topology-tls + namespace: demo +spec: + version: xpack-8.19.9 + enableSSL: true + tls: + issuerRef: + apiGroup: "cert-manager.io" + kind: Issuer + name: es-ca-issuer + topology: + master: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + data: + replicas: 2 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + ingest: + replicas: 1 + storage: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: standard + storageType: Durable + deletionPolicy: WipeOut +``` + +### Deploy Elasticsearch Topology Cluster with TLS/SSL + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/examples/elasticsearch/tls/es-topology-tls.yaml +elasticsearch.kubedb.com/es-topology-tls created +``` + +Now, wait until `es-topology-tls` has status `Ready`. i.e, + +```bash +$ kubectl get es -n demo -w +NAME VERSION STATUS AGE +es-topology-tls xpack-8.19.9 Provisioning 0s +es-topology-tls xpack-8.19.9 Provisioning 18s +. +. +es-topology-tls xpack-8.19.9 Ready 2m5s +``` + +### Verify TLS/SSL in Elasticsearch Topology Cluster + +KubeDB creates a client certificate secret for Elasticsearch. Let's check it: + +```bash +$ kubectl describe secret -n demo es-topology-tls-client-cert +Name: es-topology-tls-client-cert +Namespace: demo +Labels: app.kubernetes.io/component=database + app.kubernetes.io/instance=es-topology-tls + app.kubernetes.io/managed-by=kubedb.com + app.kubernetes.io/name=elasticsearches.kubedb.com + controller.cert-manager.io/fao=true +Annotations: cert-manager.io/alt-names: + *.es-topology-tls-pods.demo.svc,*.es-topology-tls-pods.demo.svc.cluster.local,es-topology-tls,es-topology-tls.demo.svc,localhost + cert-manager.io/certificate-name: es-topology-tls-client-cert + cert-manager.io/common-name: es-topology-tls + cert-manager.io/ip-sans: 127.0.0.1 + cert-manager.io/issuer-group: cert-manager.io + cert-manager.io/issuer-kind: Issuer + cert-manager.io/issuer-name: es-ca-issuer + cert-manager.io/subject-organizations: kubedb + cert-manager.io/uri-sans: + +Type: kubernetes.io/tls + +Data +==== +ca.crt: 1172 bytes +tls.crt: 1387 bytes +tls.key: 1704 bytes +``` + +Now, let's exec into the master node and verify the configuration that TLS is enabled for both transport and HTTP layers. + +```bash +$ kubectl exec -n demo es-topology-tls-master-0 -c elasticsearch -- \ + cat /usr/share/elasticsearch/config/elasticsearch.yml | grep -A 2 -i xpack.securit +xpack.security.enabled: true + +xpack.security.transport.ssl.enabled: true +xpack.security.transport.ssl.verification_mode: certificate +xpack.security.transport.ssl.key: certs/transport/tls.key +xpack.security.transport.ssl.certificate: certs/transport/tls.crt +xpack.security.transport.ssl.certificate_authorities: [ "certs/transport/ca.crt" ] + +xpack.security.http.ssl.enabled: true +xpack.security.http.ssl.key: certs/http/tls.key +xpack.security.http.ssl.certificate: certs/http/tls.crt +xpack.security.http.ssl.certificate_authorities: [ "certs/http/ca.crt" ] +``` + +We can see from the above output that both `xpack.security.transport.ssl.enabled: true` and `xpack.security.http.ssl.enabled: true` are set, which means TLS is enabled for both node-to-node and client-to-node communication across all topology node roles. + +Now, let's exec into the master node and connect using HTTPS to confirm the topology cluster is accessible with TLS. + +```bash +$ kubectl exec -it -n demo es-topology-tls-master-0 -c elasticsearch -- curl -k -XGET "https://localhost:9200/_cluster/health?pretty" --user "elastic:$ELASTIC_USER_PASSWORD" +{ + "cluster_name" : "es-topology-tls", + "status" : "green", + "timed_out" : false, + "number_of_nodes" : 4, + "number_of_data_nodes" : 2, + "active_primary_shards" : 1, + "active_shards" : 2, + "relocating_shards" : 0, + "initializing_shards" : 0, + "unassigned_shards" : 0, + "delayed_unassigned_shards" : 0, + "number_of_pending_tasks" : 0, + "number_of_in_flight_fetch" : 0, + "task_max_waiting_in_queue_millis" : 0, + "active_shards_percent_as_number" : 100.0 +} +``` + +From the above output, we can see that we are able to connect to the Elasticsearch topology cluster using the TLS configuration. The cluster has 4 nodes total (1 master + 2 data + 1 ingest) and is reporting `green` status. + +## Cleaning up + +To cleanup the Kubernetes resources created by this tutorial, run: + +```bash +kubectl delete es -n demo es-topology-tls +kubectl delete issuer -n demo es-ca-issuer +kubectl delete secret -n demo es-ca +kubectl delete ns demo +``` + +## Next Steps + +- Detail concepts of [Elasticsearch object](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). +- Different Elasticsearch topology clustering modes [here](/docs/guides/elasticsearch/clustering/topology-cluster/hot-warm-cold-cluster/index.md). +- Monitor your Elasticsearch database with KubeDB using [out-of-the-box Prometheus operator](/docs/guides/elasticsearch/monitoring/using-prometheus-operator.md). +- Want to hack on KubeDB? Check our [contribution guidelines](/docs/CONTRIBUTING.md). diff --git a/docs/guides/elasticsearch/tls/overview.md b/docs/guides/elasticsearch/tls/overview.md new file mode 100644 index 0000000000..6bf6fdc893 --- /dev/null +++ b/docs/guides/elasticsearch/tls/overview.md @@ -0,0 +1,70 @@ +--- +title: Elasticsearch TLS/SSL Encryption Overview +menu: + docs_{{ .version }}: + identifier: es-tls-overview + name: Overview + parent: es-tls + weight: 10 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Elasticsearch TLS/SSL Encryption + +**Prerequisite :** To configure TLS/SSL in `Elasticsearch`, `KubeDB` uses `cert-manager` to issue certificates. So first you have to make sure that the cluster has `cert-manager` installed. To install `cert-manager` in your cluster following steps [here](https://cert-manager.io/docs/installation/kubernetes/). + +To issue a certificate, the following crds of `cert-manager` are used: + +- `Issuer/ClusterIssuer`: Issuers, and ClusterIssuers represent certificate authorities (CAs) that are able to generate signed certificates by honoring certificate signing requests. All cert-manager certificates require a referenced issuer that is in a ready condition to attempt to honor the request. You can learn more details [here](https://cert-manager.io/docs/concepts/issuer/). + +- `Certificate`: `cert-manager` has the concept of Certificates that define a desired x509 certificate which will be renewed and kept up to date. You can learn more details [here](https://cert-manager.io/docs/concepts/certificate/). + +**Elasticsearch CRD Specification :** + +KubeDB uses the following crd fields to enable SSL/TLS encryption in `Elasticsearch`. + +- `spec:` + - `enableSSL` + - `tls:` + - `issuerRef` + - `certificates` + +Read about the fields in details from [elasticsearch concept](/docs/guides/elasticsearch/concepts/elasticsearch/index.md). + +When `enableSSL` is set to `true`, the users must specify the `tls.issuerRef` field. `KubeDB` uses the `issuer` or `clusterIssuer` referenced in the `tls.issuerRef` field, and the certificate specs provided in `tls.certificates` to generate certificate secrets using `Issuer/ClusterIssuer` specification. These certificate secrets including `ca.crt`, `tls.crt` and `tls.key` are used to configure both transport-layer (node-to-node) and HTTP-layer (client-to-node) TLS in `Elasticsearch`. + +## How TLS/SSL configures in Elasticsearch + +The following figure shows how `KubeDB` enterprise used to configure TLS/SSL in Elasticsearch. Open the image in a new tab to see the enlarged version. + +
+Deploy Elasticsearch with TLS/SSL +
Fig: Deploy Elasticsearch with TLS/SSL
+
+ +Deploying Elasticsearch with TLS/SSL configuration process consists of the following steps: + +1. At first, a user creates an `Issuer/ClusterIssuer` cr. + +2. Then the user creates an `Elasticsearch` CR with `enableSSL: true` which refers to the `Issuer/ClusterIssuer` CR that the user created in the previous step. + +3. `KubeDB` Provisioner operator watches for the `Elasticsearch` cr. + +4. When it finds one, it creates `Secret`, `Service`, etc. for the `Elasticsearch` cluster. + +5. `KubeDB` Ops-manager operator watches for `Elasticsearch`(5c), `Issuer/ClusterIssuer`(5b), `Secret` and `Service`(5a). + +6. When it finds all the resources(`Elasticsearch`, `Issuer/ClusterIssuer`, `Secret`, `Service`), it creates `Certificates` by using `tls.issuerRef` and `tls.certificates` field specification from `Elasticsearch` cr. + +7. `cert-manager` watches for certificates. + +8. When it finds one, it creates certificate secrets `tls-secrets` (transport and HTTP secrets etc.) that hold the actual certificate signed by the CA, containing `ca.crt`, `tls.crt` and `tls.key`. + +9. `KubeDB` Provisioner operator watches for the Certificate secrets `tls-secrets`. + +10. When it finds all the tls-secrets, it creates the related `PetSets` so that Elasticsearch can be configured with TLS/SSL for both transport and HTTP layers. + +In the next docs, we are going to show a step-by-step guide on how to configure an `Elasticsearch` cluster with TLS/SSL. diff --git a/docs/images/elasticsearch/es-reconfigure.svg b/docs/images/elasticsearch/es-reconfigure.svg new file mode 100644 index 0000000000..2fcde210e0 --- /dev/null +++ b/docs/images/elasticsearch/es-reconfigure.svg @@ -0,0 +1,110 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1. Create Elasticsearch + + Elasticsearch + + Elasticsearch + + Elasticsearch + + diff --git a/docs/images/elasticsearch/es-tls.svg b/docs/images/elasticsearch/es-tls.svg new file mode 100644 index 0000000000..074cb2c1a9 --- /dev/null +++ b/docs/images/elasticsearch/es-tls.svg @@ -0,0 +1,4 @@ + + + +            Enterprise            Operator              Community            Operator
service
se...
secret
se...
tls-secret
tls-secret
Cert- manager
Cert- ma...
StatefulSet
Statef...
Issuer/Cluster Issuer
Issuer...
Elasticsearch
Elasticsearch
Certificates
Certif...
User
User
2.Create
2.Create
1.Create
1.Create
5a.Watch
5a.Watch
3.Watch
3.Watch
4.Create
4.Create
5c.Watch
5c.Watch
6.Create
6.Create
7.Watch
7.Watch
uses
uses
8.Create
8.Create
9.Watch
9.Watch
10.Create
10.Create
5b.Watch
5b.Watch
refers to
refers toText is not SVG - cannot display \ No newline at end of file