From 898189a715af31a967f4a4cff71c626bb50e3aaa Mon Sep 17 00:00:00 2001 From: Bonusree Date: Mon, 18 May 2026 11:48:06 +0600 Subject: [PATCH 1/7] init Signed-off-by: Bonusree --- .../autoscaler/storage/cluster/index.md | 23 ++++++++++++++----- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md index fdce0ac570..a152958fa2 100644 --- a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md +++ b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md @@ -46,20 +46,29 @@ namespace/demo created ## Storage Autoscaling of Distributed Cluster Database -At first verify that your clusters have a storage class that supports volume expansion. Let's check, +At first verify that your clusters have a storage class that supports volume expansion. First let's check the storagclass of `Controler` cluster, +```bash +$ kubectl get storageclass --context demo-controller +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 4d +longhorn (default) driver.longhorn.io Delete Immediate true 24h +longhorn-static driver.longhorn.io Delete Immediate true 24h +``` +Then check the storageclass of `Worker` cluster, ```bash $ kubectl get storageclass --context demo-worker -NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE -standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 79m -topolvm-provisioner topolvm.cybozu.com Delete WaitForFirstConsumer true 78m +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 4d +longhorn (default) driver.longhorn.io Delete Immediate true 23h +longhorn-static driver.longhorn.io Delete Immediate true 23h ``` -We can see from the output the `topolvm-provisioner` storage class has `ALLOWVOLUMEEXPANSION` field as true. So, this storage class supports volume expansion. We can use it. You can install topolvm from [here](https://github.com/topolvm/topolvm) +We can see from the output the `longhorn (default)` and `longhorn-static` storage class has `ALLOWVOLUMEEXPANSION` field as true. So, this storage class supports volume expansion. We can use it. You can install `longhorn` from [here](https://longhorn.io/docs/1.11.2/deploy/install/) ### Deploy PlacementPolicy -For distributed MariaDB autoscaling, the `PlacementPolicy` must include a `monitoring.prometheus.url` for each spoke cluster. The autoscaler uses these endpoints to monitor storage usage across all clusters where MariaDB pods are running. +For distributed MariaDB autoscaling, the `PlacementPolicy` must include a `monitoring.prometheus.url` for each spoke cluster. The autoscaler uses these endpoints to monitor storage usage across all clusters where MariaDB pods are running. For storageclass you have to mention the storageclass name under both cluster like `spec.clusterSpreadConstraint.distributionRules.storageClassName` Below is the YAML of the `PlacementPolicy` that we are going to create. It distributes 4 replicas across two clusters and provides the Prometheus endpoint for each: @@ -74,6 +83,7 @@ spec: clusterSpreadConstraint: distributionRules: - clusterName: demo-controller + storageClassName: local-path monitoring: prometheus: url: http://prometheus-operated.monitoring.svc.cluster.local:9090 @@ -81,6 +91,7 @@ spec: - 0 - 2 - clusterName: demo-worker + storageClassName: local-path monitoring: prometheus: url: http://prometheus-operated.monitoring.svc.cluster.local:9090 From 663ed2df4cb8af15f109bbf49f9a6714c9d561d7 Mon Sep 17 00:00:00 2001 From: Bonusree Date: Mon, 18 May 2026 14:56:38 +0600 Subject: [PATCH 2/7] description Signed-off-by: Bonusree --- .../mariadb/distributed/autoscaler/storage/cluster/index.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md index a152958fa2..ab9c9bc388 100644 --- a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md +++ b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md @@ -68,7 +68,9 @@ We can see from the output the `longhorn (default)` and `longhorn-static` storag ### Deploy PlacementPolicy -For distributed MariaDB autoscaling, the `PlacementPolicy` must include a `monitoring.prometheus.url` for each spoke cluster. The autoscaler uses these endpoints to monitor storage usage across all clusters where MariaDB pods are running. For storageclass you have to mention the storageclass name under both cluster like `spec.clusterSpreadConstraint.distributionRules.storageClassName` +For distributed `MariaDB` autoscaling, the PlacementPolicy must define a valid `monitoring.prometheus.url` for each spoke cluster. These endpoints are required by the autoscaler to collect metrics and monitor storage usage across all clusters where MariaDB pods are deployed. + +Additionally, it is essential to explicitly specify the storage class for each cluster using `spec.clusterSpreadConstraint.distributionRules.storageClassName`. If this field is not provided, the system will fall back to the default storage class, which may lead to unintended behavior or potential data management risks in a distributed environment. Below is the YAML of the `PlacementPolicy` that we are going to create. It distributes 4 replicas across two clusters and provides the Prometheus endpoint for each: From d2a7f45a7946bfb6c66d0e0d1411fef80622b669 Mon Sep 17 00:00:00 2001 From: Bonusree Date: Wed, 20 May 2026 12:04:52 +0600 Subject: [PATCH 3/7] opsrequest Signed-off-by: Bonusree --- .../autoscaler/storage/cluster/index.md | 2 +- .../mariadb/distributed/opsrequest/index.md | 218 ++++++++++++++++++ .../mariadb/distributed/overview/index.md | 99 ++++---- 3 files changed, 270 insertions(+), 49 deletions(-) create mode 100644 docs/guides/mariadb/distributed/opsrequest/index.md diff --git a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md index ab9c9bc388..3b16841bb9 100644 --- a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md +++ b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md @@ -72,7 +72,7 @@ For distributed `MariaDB` autoscaling, the PlacementPolicy must define a valid ` Additionally, it is essential to explicitly specify the storage class for each cluster using `spec.clusterSpreadConstraint.distributionRules.storageClassName`. If this field is not provided, the system will fall back to the default storage class, which may lead to unintended behavior or potential data management risks in a distributed environment. -Below is the YAML of the `PlacementPolicy` that we are going to create. It distributes 4 replicas across two clusters and provides the Prometheus endpoint for each: +Below is the YAML of the `PlacementPolicy` that we are going to create. It distributes 3 replicas across two clusters and provides the Prometheus endpoint for each: ```yaml apiVersion: apps.k8s.appscode.com/v1 diff --git a/docs/guides/mariadb/distributed/opsrequest/index.md b/docs/guides/mariadb/distributed/opsrequest/index.md new file mode 100644 index 0000000000..0672ccfa66 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/index.md @@ -0,0 +1,218 @@ +--- +title: Distributed MariaDBOpsRequest +menu: + docs_{{ .version }}: + identifier: guides-mariadb-distributed-opsrequest + name: Distributed MariaDBOpsRequest + parent: guides-mariadb-distributed + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Opsrequest of a Distributed MariaDB Cluster +This guide demonstrates how to use `OpsRequest` with the `MariaDB` CRD in a distributed cluster setup. + +> **Note:** All `OpsRequest` operations behave consistently in a distributed environment, except for `VolumeExpansion`, `HorizontalScaling`, and `Reconfigure`, which have cluster-specific considerations. + + +## Before You Begin + +- At first, you need to have a multi-cluster Kubernetes setup with OCM and KubeSlice configured. Follow the [Distributed MariaDB Overview](/docs/guides/mariadb/distributed/overview) guide to set up the required infrastructure. + +- Install `KubeDB` Community, Ops-Manager and Autoscaler operator in your hub cluster following the steps [here](/docs/setup/README.md). Make sure to enable OCM support: + +```bash + --set petset.features.ocm.enabled=true +``` + +- Install `Metrics Server` from [here](https://github.com/kubernetes-sigs/metrics-server#installation) + +- Install Prometheus in each spoke cluster. The autoscaler queries the per-cluster Prometheus endpoint (configured in the `PlacementPolicy`) to collect resource usage metrics. You can install it from [here](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) + +- You should be familiar with the following `KubeDB` concepts: + - [MariaDB](/docs/guides/mariadb/concepts/mariadb) + - [MariaDBDistributedOverview](/docs/guides/mariadb/distributed/overview/index.md) + - [MariaDBOpsRequest](/docs/guides/mariadb/concepts/opsrequest) +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +## OpsRequest of Distributed Cluster Database + +Here, we are going to deploy a distributed `MariaDB` Galera cluster using a supported version by `KubeDB` operator. Then we are going to apply `MariaDopsRequest` to set up autoscaling. + +### Deploy PlacementPolicy +For horizontal scaling, you can increase or decrease the number of replicas as needed. To support this, the `PlacementPolicy` should be defined with the maximum number of replicas you anticipate requiring. This allows you to scale up to that limit while also permitting operation with fewer replicas if needed. + +In this example, the `PlacementPolicy` is configured with a maximum of five replicas. It also explicitly defines how replicas are distributed across clusters, specifying which replicas are assigned to each cluster. + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + labels: + app.kubernetes.io/managed-by: Helm + name: distributed-mariadb +spec: + clusterSpreadConstraint: + distributionRules: + - clusterName: demo-controller + replicaIndices: + - 0 + - 2 + - clusterName: demo-worker + replicaIndices: + - 1 + slice: + projectNamespace: kubeslice-demo-distributed-mariadb + sliceName: demo-slice + nodeSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway + zoneSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway +``` + +Here, + +- `spec.clusterSpreadConstraint.distributionRules[].replicaIndices` specifies which MariaDB replica indices are scheduled on that cluster. Here `demo-controller` hosts replicas `0` and `2`, and `demo-worker` hosts replica `1`. + +Apply the `PlacementPolicy` on the hub (`demo-controller`) cluster: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/autoscaler/compute/cluster/examples/placement-policy.yaml --context demo-controller +placementpolicy.apps.k8s.appscode.com/distributed-mariadb created +``` + +### Deploy Distributed MariaDB Cluster + +In this section, we are going to deploy a distributed MariaDB Galera cluster with version `11.5.2`. Then, in the next section we will set up autoscaling for this database using `MariaDBAutoscaler` CRD. + +Below is the YAML of the `MariaDB` CR that we are going to create. Note that `spec.distributed` is set to `true` and the `PlacementPolicy` is referenced via `spec.podTemplate.spec.podPlacementPolicy`: + +```yaml +apiVersion: kubedb.com/v1 +kind: MariaDB +metadata: + name: sample-mariadb + namespace: demo +spec: + version: "11.5.2" + distributed: true + replicas: 3 + storageType: Durable + storage: + storageClassName: "longhorn" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + podTemplate: + spec: + podPlacementPolicy: + name: distributed-mariadb + containers: + - name: mariadb + resources: + requests: + cpu: "200m" + memory: "300Mi" + limits: + cpu: "200m" + memory: "300Mi" + deletionPolicy: WipeOut +``` + +Let's create the `MariaDB` CRO we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/autoscaler/compute/cluster/examples/sample-mariadb.yaml --context demo-controller +mariadb.kubedb.com/sample-mariadb created +``` + +Now, wait until `sample-mariadb` has status `Ready`. i.e, + +```bash +$ kubectl get mariadb -n demo --context demo-controller +NAME VERSION STATUS AGE +sample-mariadb 11.5.2 Ready 14m +``` + +The pods are distributed across clusters as defined by the `PlacementPolicy`: + +```bash +$ kubectl get pod -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 14m +mariadb-1 3/3 Running 0 14m + +$ kubectl get pod -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-2 3/3 Running 0 15m + + +## Scale Up Replicas + +Here, we are going to scale up the replicas of the replicaset to meet the desired number of replicas after scaling. + +#### Create MariaDBOpsRequest + +In order to scale up the replicas of the replicaset of the database, we have to create a `MariaDBOpsRequest` CR with our desired replicas. Below is the YAML of the `MariaDBOpsRequest` CR that we are going to create, + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-up + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member : 5 +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing horizontal scaling operation on `sample-mariadb` database. +- `spec.type` specifies that we are performing `HorizontalScaling` on our database. +- `spec.horizontalScaling.member` specifies the desired replicas after scaling. + +Let's create the `MariaDBOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/scaling/horizontal-scaling/cluster/example/mdops-upscale.yaml +mariadbopsrequest.ops.kubedb.com/mdops-scale-horizontal-up created +``` + +#### Verify Cluster replicas scaled up successfully + +If everything goes well, `KubeDB` Enterprise operator will update the replicas of `MariaDB` object and related `PetSets` and `Pods`. + +Let's wait for `MariaDBOpsRequest` to be `Successful`. Run the following command to watch `MariaDBOpsRequest` CR, + +```bash +$ watch kubectl get mariadbopsrequest -n demo +Every 2.0s: kubectl get mariadbopsrequest -n demo +NAME TYPE STATUS AGE +mdps-scale-horizontal HorizontalScaling Successful 106s +``` + +We can see from the above output that the `MariaDBOpsRequest` has succeeded. Now, we are going to verify the number of replicas this database has from the MariaDB object, number of pods the petset have, + +```bash +$ kubectl get mariadb -n demo sample-mariadb -o json | jq '.spec.replicas' +5 +$ kubectl get sts -n demo sample-mariadb -o json | jq '.spec.replicas' +5 +``` + diff --git a/docs/guides/mariadb/distributed/overview/index.md b/docs/guides/mariadb/distributed/overview/index.md index f2ffe93a07..92250d4841 100644 --- a/docs/guides/mariadb/distributed/overview/index.md +++ b/docs/guides/mariadb/distributed/overview/index.md @@ -51,12 +51,12 @@ Follow these steps to deploy a distributed MariaDB Galera cluster across multipl Ensure your `KUBECONFIG` is set up to switch between clusters. This guide uses two clusters: `demo-controller` (hub and spoke) and `demo-worker` (spoke). ```bash -kubectl config get-contexts +$ kubectl config get-contexts ``` **Output:** -``` +```bash CURRENT NAME CLUSTER AUTHINFO NAMESPACE * demo-controller demo-controller demo-controller demo-worker demo-worker demo-worker @@ -67,8 +67,8 @@ CURRENT NAME CLUSTER AUTHINFO NAMESPACE On the `demo-controller` cluster, initialize the OCM hub: ```bash -kubectl config use-context demo-controller -clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true +$ kubectl config use-context demo-controller +$ clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true ``` #### 3. Verify Hub Deployment @@ -76,7 +76,7 @@ clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true Check the pods in the `open-cluster-management-hub` namespace to ensure all components are running: ```bash -kubectl get pods -n open-cluster-management-hub +$ kubectl get pods -n open-cluster-management-hub ``` **Output:** @@ -98,7 +98,7 @@ All pods should be in the `Running` state with `1/1` readiness and no restarts, Obtain the join token from the hub cluster: ```bash -clusteradm get token +$ clusteradm get token ``` **Output:** @@ -112,8 +112,8 @@ clusteradm join --hub-token --hub-apiserver https:/ On the `demo-worker` cluster, join it to the hub. Include the `RawFeedbackJsonString` feature gate for resource feedback: ```bash -kubectl config use-context demo-worker -clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true +$ kubectl config use-context demo-worker +$ clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true ``` #### 5. Accept Spoke Cluster @@ -121,8 +121,8 @@ clusteradm join --hub-token --hub-apiserver https:/ On the `demo-controller` cluster, accept the `demo-worker` cluster: ```bash -kubectl config use-context demo-controller -clusteradm accept --clusters demo-worker +$ kubectl config use-context demo-controller +$ clusteradm accept --clusters demo-worker ``` > **Note:** It may take a few attempts (e.g., retry every 10 seconds) if the cluster is not immediately available. @@ -141,12 +141,12 @@ Your managed cluster demo-worker has joined the Hub successfully. Confirm that a namespace for `demo-worker` was created on the hub cluster: ```bash -kubectl get ns +$ kubectl get ns ``` **Output:** -``` +```bash NAME STATUS AGE default Active 99m demo-worker Active 58s @@ -162,15 +162,15 @@ open-cluster-management-hub Active 5m32s Repeat the join and accept process for `demo-controller` so it can also act as a spoke cluster: ```bash -kubectl config use-context demo-controller -clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true -clusteradm accept --clusters demo-controller +$ kubectl config use-context demo-controller +$ clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true +$ clusteradm accept --clusters demo-controller ``` Verify the namespace for `demo-controller`: ```bash -kubectl get ns +$ kubectl get ns ``` **Output:** @@ -194,7 +194,7 @@ open-cluster-management-hub Active 10m If you did not follow the provided OCM installation steps, update the `klusterlet` resource to enable feedback retrieval: ```bash -kubectl edit klusterlet klusterlet +$ kubectl edit klusterlet klusterlet ``` Add the following under the `spec` field: @@ -213,7 +213,7 @@ workConfiguration: Verify the configuration: ```bash -kubectl get klusterlet klusterlet -oyaml +$ kubectl get klusterlet klusterlet -oyaml ``` **Sample Output (abridged):** @@ -255,7 +255,7 @@ kubeslice: Deploy the controller using Helm: ```bash -helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ +$ helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ --version v2026.1.15 \ -f controller.yaml \ --namespace kubeslice-controller \ @@ -296,18 +296,18 @@ spec: Apply the project: ```bash -kubectl apply -f project.yaml +$ kubectl apply -f project.yaml ``` Verify: ```bash -kubectl get project -n kubeslice-controller +$ kubectl get project -n kubeslice-controller ``` **Output:** -``` +```bash NAME AGE demo-distributed-mariadb 31s ``` @@ -315,7 +315,7 @@ demo-distributed-mariadb 31s Check service accounts: ```bash -kubectl get sa -n kubeslice-demo-distributed-mariadb +$ kubectl get sa -n kubeslice-demo-distributed-mariadb ``` **Output:** @@ -333,16 +333,16 @@ Assign the `kubeslice.io/node-type=gateway` label to the node where the worker o On `demo-controller`: ```bash -kubectl get nodes -kubectl label node demo-master kubeslice.io/node-type=gateway +$ kubectl get nodes +$ kubectl label node demo-master kubeslice.io/node-type=gateway ``` On `demo-worker`: ```bash -kubectl config use-context demo-worker -kubectl get nodes -kubectl label node demo-worker kubeslice.io/node-type=gateway +$ kubectl config use-context demo-worker +$ kubectl get nodes +$ kubectl label node demo-worker kubeslice.io/node-type=gateway ``` #### 4. Register Clusters with KubeSlice @@ -414,13 +414,13 @@ spec: Apply on `demo-controller`: ```bash -kubectl apply -f registration.yaml +$ kubectl apply -f registration.yaml ``` Verify: ```bash -kubectl get clusters -n kubeslice-demo-distributed-mariadb +$ kubectl get clusters -n kubeslice-demo-distributed-mariadb ``` **Output:** @@ -434,7 +434,7 @@ demo-worker 9s Verify the worker installation: ```bash -kubectl get pods -n kubeslice-system +$ kubectl get pods -n kubeslice-system ``` **Output:** @@ -503,7 +503,7 @@ spec: Apply the `SliceConfig`: ```bash -kubectl apply -f sliceconfig.yaml +$ kubectl apply -f sliceconfig.yaml ``` #### 6. Configure DNS for KubeSlice @@ -513,12 +513,12 @@ Update the network traffic rules to forward the `*.slice.local` suffix (KubeSlic First, get the KubeSlice DNS service IP address: ```bash -kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' +$ kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' ``` **Output:** -``` +```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubeslice-dns ClusterIP 10.43.172.191 53/UDP,53/TCP 8d app=kubeslice-dns ``` @@ -527,7 +527,7 @@ Traffic for `.slice.local` should be forwarded to `10.43.172.191` for this clust Add the following section to your CoreDNS configuration: -``` +```bash slice.local:53 { errors cache 30 @@ -538,7 +538,7 @@ slice.local:53 { Example CoreDNS ConfigMap: ```bash -kubectl get cm -n kube-system coredns -oyaml +$ kubectl get cm -n kube-system coredns -oyaml ``` **Output:** @@ -593,7 +593,7 @@ Install the KubeDB Operator on the `demo-controller` cluster to manage the Maria Download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb). Obtain the license for the `demo-controller` cluster. ```bash -helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ +$ helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ --version v2026.2.26 \ --namespace kubedb --create-namespace \ --set-file global.license=$HOME/Downloads/kubedb-license-cd548cce-5141-4ed3-9276-6d9578707f12.txt \ @@ -608,12 +608,12 @@ For additional details, refer to the [KubeDB Installation Guide](https://kubedb. Verify that the pods are running: ```bash -kubectl get pods -n kubedb +$ kubectl get pods -n kubedb ``` **Output:** -``` +```bash NAME READY STATUS RESTARTS AGE kubedb-kubedb-autoscaler-0 1/2 Running 0 44s kubedb-kubedb-ops-manager-0 1/2 Running 0 43s @@ -624,8 +624,9 @@ kubedb-sidekick-5dbf7bcf64-4b8cw 2/2 Running 0 44s ``` ### Step 5: Define a PlacementPolicy +You can define the `storageClassName` under `spec.clusterSpreadConstraint.distributionRules` for each cluster. If not explicitly specified, the clusters will automatically select an appropriate storage class. Additionally, you can control replica placement by defining which replicas belong to which cluster. When scaling the number of replicas, the distribution must also be specified at the cluster level. In this example, we are using three replicas. -Create a `PlacementPolicy` to control pod distribution across clusters. Create a `pod-placement-policy.yaml` file: +To manage pod distribution across clusters, create a `PlacementPolicy`. For this purpose, define a `pod-placement-policy.yaml` file as shown below: ```yaml apiVersion: apps.k8s.appscode.com/v1 @@ -638,10 +639,12 @@ spec: clusterSpreadConstraint: distributionRules: - clusterName: demo-controller + storageClassName: local-path replicaIndices: - 0 - 2 - clusterName: demo-worker + storageClassName: local-path replicaIndices: - 1 slice: @@ -663,7 +666,7 @@ This policy schedules: Apply the policy on `demo-controller`: ```bash -kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 6: Create a Distributed MariaDB Instance @@ -697,7 +700,7 @@ spec: Apply the resource on `demo-controller`: ```bash -kubectl apply -f mariadb.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f mariadb.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 7: Verify the Deployment @@ -705,12 +708,12 @@ kubectl apply -f mariadb.yaml --context demo-controller --kubeconfig $HOME/.kube #### 1. Check MariaDB Resource and Pods on `demo-controller` ```bash -kubectl get md,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl get md,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config ``` **Output:** -``` +```bash NAME VERSION STATUS AGE mariadb.kubedb.com/mariadb 11.5.2 Ready 99s @@ -725,12 +728,12 @@ secret/mariadb-auth kubernetes.io/basic-auth 2 95s #### 2. Check Pods and Secrets on `demo-worker` ```bash -kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config +$ kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config ``` **Output:** -``` +```bash NAME READY STATUS RESTARTS AGE mariadb-1 3/3 Running 0 95s @@ -743,7 +746,7 @@ secret/mariadb-auth kubernetes.io/basic-auth 2 95s Connect to a MariaDB pod and check the Galera cluster status. The primary service DNS follows the format `..svc`: ```bash -kubectl exec -it -n demo pod/mariadb-0 --context demo-controller -- bash +$ kubectl exec -it -n demo pod/mariadb-0 --context demo-controller -- bash mariadb -uroot -p$MYSQL_ROOT_PASSWORD -hmariadb.demo.svc ``` @@ -755,7 +758,7 @@ SHOW STATUS LIKE 'wsrep_cluster_status'; **Output:** -``` +```bash +----------------------+---------+ | Variable_name | Value | +----------------------+---------+ From 3d5538610c15b881bea0938103cb367daa54140e Mon Sep 17 00:00:00 2001 From: Bonusree Date: Wed, 10 Jun 2026 16:57:57 +0600 Subject: [PATCH 4/7] some changes in configuration file Signed-off-by: Bonusree --- .../mariadb/distributed/overview/index.md | 169 +++++++++++------- 1 file changed, 104 insertions(+), 65 deletions(-) diff --git a/docs/guides/mariadb/distributed/overview/index.md b/docs/guides/mariadb/distributed/overview/index.md index 92250d4841..34e8fe0f01 100644 --- a/docs/guides/mariadb/distributed/overview/index.md +++ b/docs/guides/mariadb/distributed/overview/index.md @@ -106,14 +106,14 @@ $ clusteradm get token ``` token= please log on spoke and run: -clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name +clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name ``` On the `demo-worker` cluster, join it to the hub. Include the `RawFeedbackJsonString` feature gate for resource feedback: ```bash $ kubectl config use-context demo-worker -$ clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true ``` #### 5. Accept Spoke Cluster @@ -163,7 +163,7 @@ Repeat the join and accept process for `demo-controller` so it can also act as a ```bash $ kubectl config use-context demo-controller -$ clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true $ clusteradm accept --clusters demo-controller ``` @@ -189,25 +189,44 @@ open-cluster-management-agent-addon Active 34s open-cluster-management-hub Active 10m ``` -### Step 2: Configure OCM WorkConfiguration (Optional) +#### 8. Verify OCM Roles -If you did not follow the provided OCM installation steps, update the `klusterlet` resource to enable feedback retrieval: +After registration, use these commands to confirm which cluster is the hub and which are spokes: ```bash -$ kubectl edit klusterlet klusterlet +# Hub: lists all registered spoke clusters +$ kubectl get managedclusters + +# Spoke: shows this cluster's registered name +$ kubectl get klusterlet klusterlet -o jsonpath='{.spec.clusterName}' + +# Hub components run only on the hub cluster +$ kubectl get pods -n open-cluster-management-hub + +# Spoke agent runs on every spoke cluster +$ kubectl get pods -n open-cluster-management-agent ``` -Add the following under the `spec` field: +### Step 2: Configure OCM WorkConfiguration -```yaml -workConfiguration: - featureGates: - - feature: RawFeedbackJsonString - mode: Enable - hubKubeAPIBurst: 100 - hubKubeAPIQPS: 50 - kubeAPIBurst: 100 - kubeAPIQPS: 50 +Run this on **every spoke cluster** (`demo-controller` and `demo-worker`). This enables the `RawFeedbackJsonString` feature gate that KubeDB requires to read pod status across clusters, and raises the API rate limits to prevent throttling. + +> **Note:** Even if you passed `--feature-gates=RawFeedbackJsonString=true` during `clusteradm join`, the rate limit fields are not set by that flag. Run this patch on all spokes regardless. +> +> **Why this matters:** KubeDB uses OCM's ManifestWork feedback mechanism to watch the status of MariaDB pods on remote spoke clusters. Without `RawFeedbackJsonString`, the KubeDB provisioner on the hub never receives pod status updates from spokes and the distributed MariaDB CR will stay in a non-Ready state indefinitely. The rate limits prevent the klusterlet agent from being API-throttled during initial cluster formation. + +```bash +$ kubectl patch klusterlet klusterlet --type=merge -p '{ + "spec": { + "workConfiguration": { + "featureGates": [{"feature": "RawFeedbackJsonString", "mode": "Enable"}], + "hubKubeAPIBurst": 100, + "hubKubeAPIQPS": 50, + "kubeAPIBurst": 100, + "kubeAPIQPS": 50 + } + } +}' ``` Verify the configuration: @@ -241,7 +260,13 @@ KubeSlice enables pod-to-pod communication across clusters. Install the KubeSlic #### 1. Install KubeSlice Controller -On `demo-controller`, create a `controller.yaml` file: +On `demo-controller`, get the hub API server address first: + +```bash +$ kubectl cluster-info | grep 'Kubernetes control plane' +``` + +Use the IP and port from that output as the `endpoint` value. Create a `controller.yaml` file: ```yaml kubeslice: @@ -249,7 +274,7 @@ kubeslice: loglevel: info rbacResourcePrefix: kubeslice-rbac projectnsPrefix: kubeslice - endpoint: https://10.2.0.56:6443 + endpoint: https://:6443 ``` Deploy the controller using Helm: @@ -267,7 +292,7 @@ $ helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-c Verify the installation: ```bash -kubectl get pods -n kubeslice-controller +$ kubectl get pods -n kubeslice-controller ``` **Output:** @@ -334,7 +359,7 @@ On `demo-controller`: ```bash $ kubectl get nodes -$ kubectl label node demo-master kubeslice.io/node-type=gateway +$ kubectl label node kubeslice.io/node-type=gateway ``` On `demo-worker`: @@ -342,15 +367,15 @@ On `demo-worker`: ```bash $ kubectl config use-context demo-worker $ kubectl get nodes -$ kubectl label node demo-worker kubeslice.io/node-type=gateway +$ kubectl label node kubeslice.io/node-type=gateway ``` #### 4. Register Clusters with KubeSlice -Identify the network interface for both clusters by running the following command on the node: +Identify the network interface for each cluster by running the following command **on the gateway node of each cluster**: ```bash -ip route get 8.8.8.8 | awk '{ print $5 }' +$ ip route get 8.8.8.8 | awk '{ print $5 }' ``` **Output (example):** @@ -359,11 +384,15 @@ ip route get 8.8.8.8 | awk '{ print $5 }' enp1s0 ``` +> **Important:** The `networkInterface` value must match the primary network interface of each cluster's gateway node. Run the command above on **each cluster separately** and use that cluster's output as its `networkInterface` value. Each cluster may have a **different interface name** — do not assume they are the same. Using the wrong interface name will cause the WireGuard gateway to silently fail and cross-cluster MariaDB replication will never connect. +> +> Example: if `demo-controller` returns `enp3s0` and `demo-worker` returns `eth0`, use those exact values in the YAML below. + Create a `registration.yaml` file: -> **Important:** +> **Note:** > - The cluster name must exactly match the name of the OCM (spoke) cluster. -> - The corresponding `ManagedClusterAddOn` resource must be created in the namespace that bears the same name as the cluster to setup kubeslice worker automatically. +> - The corresponding `ManagedClusterAddOn` resource must be created in the namespace that bears the same name as the cluster to set up the KubeSlice worker automatically. ```yaml apiVersion: controller.kubeslice.io/v1alpha1 @@ -372,7 +401,7 @@ metadata: name: demo-controller namespace: kubeslice-demo-distributed-mariadb spec: - networkInterface: enp1s0 + networkInterface: # replace with output of ip route command clusterProperty: {} --- apiVersion: controller.kubeslice.io/v1alpha1 @@ -381,7 +410,7 @@ metadata: name: demo-worker namespace: kubeslice-demo-distributed-mariadb spec: - networkInterface: enp1s0 + networkInterface: # replace with output of ip route command clusterProperty: {} --- apiVersion: addon.open-cluster-management.io/v1alpha1 @@ -417,27 +446,28 @@ Apply on `demo-controller`: $ kubectl apply -f registration.yaml ``` -Verify: +Verify OCM is deploying the KubeSlice worker manifests to each cluster: ```bash -$ kubectl get clusters -n kubeslice-demo-distributed-mariadb +$ kubectl get managedclusteraddon -A ``` **Output:** ``` -NAME AGE -demo-controller 9s -demo-worker 9s +NAMESPACE NAME AVAILABLE DEGRADED PROGRESSING +demo-controller kubeslice Unknown True +demo-worker kubeslice Unknown True ``` -Verify the worker installation: +`PROGRESSING: True` means OCM is actively deploying. Wait until `kubeslice-operator` shows `2/2 Running` on both clusters before proceeding: ```bash -$ kubectl get pods -n kubeslice-system +# Run on each spoke cluster +$ kubectl get pods -n kubeslice-system --watch ``` -**Output:** +**Expected output (after KubeSlice worker is fully deployed):** ``` NAME READY STATUS RESTARTS AGE @@ -446,8 +476,6 @@ kubeslice-dns-6bd9749f4d-pvh7g 1/1 Running 0 4m43s kubeslice-install-crds-szhvc 0/1 Completed 0 4m56s kubeslice-netop-g4dfn 1/1 Running 0 4m43s kubeslice-operator-949b7d6f7-9wj7h 2/2 Running 0 4m43s -kubeslice-postdelete-job-ctlzt 0/1 Completed 0 20m -nsm-delete-webhooks-ndksl 0/1 Completed 0 20m nsc-grpc-server-sbjj7 1/1 Running 0 4m43s nsm-install-crds-5z4j9 0/1 Completed 0 4m53s nsmgr-zzwgh 2/2 Running 0 4m43s @@ -506,11 +534,15 @@ Apply the `SliceConfig`: $ kubectl apply -f sliceconfig.yaml ``` +After the SliceConfig is applied, a `vl3-slice-router` pod will appear in `kubeslice-system` on each cluster, indicating the slice VPN tunnel is being established. + #### 6. Configure DNS for KubeSlice -Update the network traffic rules to forward the `*.slice.local` suffix (KubeSlice default domain) to the KubeSlice gateway. +Update CoreDNS to forward `*.slice.local` traffic to the KubeSlice DNS service. Run the following steps on **every cluster** in the slice. + +> **Important:** CoreDNS must be updated and restarted on all clusters before proceeding to Step 4 (KubeDB install). Galera nodes use `.slice.local` DNS names to discover each other across clusters. If DNS is not configured before MariaDB pods start, replication will not form. -First, get the KubeSlice DNS service IP address: +Get the KubeSlice DNS service IP address on each cluster: ```bash $ kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' @@ -523,11 +555,9 @@ NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubeslice-dns ClusterIP 10.43.172.191 53/UDP,53/TCP 8d app=kubeslice-dns ``` -Traffic for `.slice.local` should be forwarded to `10.43.172.191` for this cluster. +Add the following block to the **top** of your CoreDNS `Corefile` ConfigMap (replace the IP with the one from your cluster): -Add the following section to your CoreDNS configuration: - -```bash +``` slice.local:53 { errors cache 30 @@ -535,7 +565,7 @@ slice.local:53 { } ``` -Example CoreDNS ConfigMap: +Example of the full CoreDNS ConfigMap after editing: ```bash $ kubectl get cm -n kube-system coredns -oyaml @@ -582,28 +612,44 @@ metadata: namespace: kube-system ``` -> **Note:** Follow the same procedure to update the DNS configuration for all clusters in the slice. +After editing the ConfigMap, restart CoreDNS to apply the change: + +```bash +$ kubectl rollout restart deploy/coredns -n kube-system +``` + +Repeat the DNS configuration steps on every cluster in the slice. ### Step 4: Install the KubeDB Operator -Install the KubeDB Operator on the `demo-controller` cluster to manage the MariaDB instance. +> **Note:** Install the KubeDB operator only on the hub cluster (`demo-controller`). The operator manages MariaDB pods on spoke clusters through OCM — no KubeDB installation is needed on `demo-worker`. #### Get a Free License -Download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb). Obtain the license for the `demo-controller` cluster. +The KubeDB license is tied to the `kube-system` namespace UID of the hub cluster and has an expiry date. Get your cluster UID and verify the license before installing: + +```bash +# Get your cluster UID (required when requesting the license) +$ kubectl get ns kube-system -o jsonpath='{.metadata.uid}' + +# Verify the license is not expired +$ openssl x509 -noout -enddate -in $HOME/Downloads/kubedb-license-.txt +``` + +If expired or not yet obtained, download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb) using the cluster UID above. ```bash $ helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ --version v2026.2.26 \ --namespace kubedb --create-namespace \ - --set-file global.license=$HOME/Downloads/kubedb-license-cd548cce-5141-4ed3-9276-6d9578707f12.txt \ + --set-file global.license=$HOME/Downloads/kubedb-license-.txt \ --set petset.features.ocm.enabled=true \ --wait --burst-limit=10000 --debug ``` > **Note:** The `--set petset.features.ocm.enabled=true` flag must be set to enable the MariaDB Distributed feature. -For additional details, refer to the [KubeDB Installation Guide](https://kubedb.com/docs/v2025.6.30/setup/install/kubedb/). +For additional details, refer to the [KubeDB Installation Guide](https://kubedb.com/docs/v2026.4.27/setup/). Verify that the pods are running: @@ -624,6 +670,7 @@ kubedb-sidekick-5dbf7bcf64-4b8cw 2/2 Running 0 44s ``` ### Step 5: Define a PlacementPolicy + You can define the `storageClassName` under `spec.clusterSpreadConstraint.distributionRules` for each cluster. If not explicitly specified, the clusters will automatically select an appropriate storage class. Additionally, you can control replica placement by defining which replicas belong to which cluster. When scaling the number of replicas, the distribution must also be specified at the cluster level. In this example, we are using three replicas. To manage pod distribution across clusters, create a `PlacementPolicy`. For this purpose, define a `pod-placement-policy.yaml` file as shown below: @@ -639,12 +686,12 @@ spec: clusterSpreadConstraint: distributionRules: - clusterName: demo-controller - storageClassName: local-path + storageClassName: local-path # optional; omit to use the cluster's default storage class replicaIndices: - 0 - 2 - clusterName: demo-worker - storageClassName: local-path + storageClassName: local-path # optional; omit to use the cluster's default storage class replicaIndices: - 1 slice: @@ -671,6 +718,12 @@ $ kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubecon ### Step 6: Create a Distributed MariaDB Instance +Create the `demo` namespace first: + +```bash +$ kubectl create namespace demo +``` + Define a MariaDB custom resource with `spec.distributed` set to `true` and reference the `PlacementPolicy`. Create a `mariadb.yaml` file: ```yaml @@ -773,20 +826,7 @@ Check additional Galera status variables: SHOW STATUS LIKE 'wsrep%'; ``` -**Key Indicators:** - -- `wsrep_cluster_status: Primary` — The cluster is fully operational. -- `wsrep_cluster_size: 3` — All three nodes are part of the cluster. -- `wsrep_connected: ON` — The node is connected to the cluster. -- `wsrep_ready: ON` — The node is ready to accept queries. -- `wsrep_incoming_addresses` — Lists the IP addresses of all nodes (e.g., `10.1.0.3:0,10.1.0.4:0,10.1.16.4:0`). -## Troubleshooting - -- **Pods Not Running**: Check pod logs (`kubectl logs -n demo mariadb-0`) for errors related to storage, networking, or configuration. -- **Cluster Not Joining**: Ensure the `RawFeedbackJsonString` feature gate is enabled and verify network connectivity between clusters. -- **KubeSlice Issues**: Confirm that the network interface (e.g., `enp1s0`) matches your cluster's configuration and that sidecar containers are injected. -- **MariaDB Not Synced**: Check `wsrep_local_state_comment` (should be `Synced`) and ensure all nodes have the same `wsrep_cluster_state_uuid`. ## Next Steps @@ -795,4 +835,3 @@ SHOW STATUS LIKE 'wsrep%'; - **Monitoring**: Integrate KubeDB with monitoring tools like Prometheus for cluster health insights. For further details, refer to the [KubeDB Documentation](https://kubedb.com/docs/v2025.7.31/). - From 969f0e80105267ea54df34b54ec723812fba19ee Mon Sep 17 00:00:00 2001 From: Bonusree Date: Fri, 12 Jun 2026 17:49:54 +0600 Subject: [PATCH 5/7] horizontal scale Signed-off-by: Bonusree --- .../mariadb/distributed/opsrequest/_index.md | 10 + .../mariadb/distributed/opsrequest/index.md | 249 +++- .../postgres/distributed/overview/index.md | 1139 +++++++++-------- 3 files changed, 819 insertions(+), 579 deletions(-) create mode 100644 docs/guides/mariadb/distributed/opsrequest/_index.md diff --git a/docs/guides/mariadb/distributed/opsrequest/_index.md b/docs/guides/mariadb/distributed/opsrequest/_index.md new file mode 100644 index 0000000000..cef45b7cf9 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/_index.md @@ -0,0 +1,10 @@ +--- +title: Horizontal Scaling +menu: + docs_{{ .version }}: + identifier: guides-mariadb-distributed-horizontalscaling + name: Autoscaling + parent: guides-mariadb-distributed + weight: 47 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/mariadb/distributed/opsrequest/index.md b/docs/guides/mariadb/distributed/opsrequest/index.md index 0672ccfa66..ae4f0052b9 100644 --- a/docs/guides/mariadb/distributed/opsrequest/index.md +++ b/docs/guides/mariadb/distributed/opsrequest/index.md @@ -4,7 +4,7 @@ menu: docs_{{ .version }}: identifier: guides-mariadb-distributed-opsrequest name: Distributed MariaDBOpsRequest - parent: guides-mariadb-distributed + parent: guides-mariadb-distributed-horizontalscaling weight: 20 menu_name: docs_{{ .version }} section_menu_id: guides @@ -12,30 +12,28 @@ section_menu_id: guides > New to KubeDB? Please start [here](/docs/README.md). -# Opsrequest of a Distributed MariaDB Cluster -This guide demonstrates how to use `OpsRequest` with the `MariaDB` CRD in a distributed cluster setup. +# Horizontal Scale Distributed MariaDB -> **Note:** All `OpsRequest` operations behave consistently in a distributed environment, except for `VolumeExpansion`, `HorizontalScaling`, and `Reconfigure`, which have cluster-specific considerations. +This guide will show you how to use `KubeDB` Enterprise operator to horizontally scale a distributed MariaDB Galera cluster across multiple Kubernetes clusters. +> **Note:** All other `OpsRequest` operations behave consistently in a distributed environment. Only `HorizontalScaling` has cluster-specific considerations covered in this guide. ## Before You Begin - At first, you need to have a multi-cluster Kubernetes setup with OCM and KubeSlice configured. Follow the [Distributed MariaDB Overview](/docs/guides/mariadb/distributed/overview) guide to set up the required infrastructure. -- Install `KubeDB` Community, Ops-Manager and Autoscaler operator in your hub cluster following the steps [here](/docs/setup/README.md). Make sure to enable OCM support: +- Install `KubeDB` Community and Enterprise operator in your hub cluster following the steps [here](/docs/setup/README.md). Make sure to enable OCM support: -```bash + ```bash --set petset.features.ocm.enabled=true -``` - -- Install `Metrics Server` from [here](https://github.com/kubernetes-sigs/metrics-server#installation) - -- Install Prometheus in each spoke cluster. The autoscaler queries the per-cluster Prometheus endpoint (configured in the `PlacementPolicy`) to collect resource usage metrics. You can install it from [here](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) + ``` - You should be familiar with the following `KubeDB` concepts: - - [MariaDB](/docs/guides/mariadb/concepts/mariadb) - - [MariaDBDistributedOverview](/docs/guides/mariadb/distributed/overview/index.md) - - [MariaDBOpsRequest](/docs/guides/mariadb/concepts/opsrequest) + - [MariaDB](/docs/guides/mariadb/concepts/mariadb/) + - [Distributed MariaDB Overview](/docs/guides/mariadb/distributed/overview/index.md) + - [MariaDBOpsRequest](/docs/guides/mariadb/concepts/opsrequest/) + - [Horizontal Scaling Overview](/docs/guides/mariadb/scaling/horizontal-scaling/overview/) + To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. ```bash @@ -43,14 +41,15 @@ $ kubectl create ns demo namespace/demo created ``` -## OpsRequest of Distributed Cluster Database +## Apply Horizontal Scaling on Distributed Cluster -Here, we are going to deploy a distributed `MariaDB` Galera cluster using a supported version by `KubeDB` operator. Then we are going to apply `MariaDopsRequest` to set up autoscaling. +Here, we are going to deploy a distributed `MariaDB` Galera cluster and then apply horizontal scaling using `MariaDBOpsRequest`. ### Deploy PlacementPolicy -For horizontal scaling, you can increase or decrease the number of replicas as needed. To support this, the `PlacementPolicy` should be defined with the maximum number of replicas you anticipate requiring. This allows you to scale up to that limit while also permitting operation with fewer replicas if needed. -In this example, the `PlacementPolicy` is configured with a maximum of five replicas. It also explicitly defines how replicas are distributed across clusters, specifying which replicas are assigned to each cluster. +For a distributed MariaDB cluster, the `PlacementPolicy` must be created **before** deploying the database. It defines which replica index is scheduled on which cluster and acts as the upper bound for scaling — the actual running replicas can be any number less than or equal to the total indices defined in the policy. + +In this example, the `PlacementPolicy` is configured with **five** replica indices distributed across two clusters. The MariaDB cluster will start with **three** replicas (indices `0`, `1`, `2`), and can be scaled up to **five** (indices `0`–`4`) without modifying the policy. ```yaml apiVersion: apps.k8s.appscode.com/v1 @@ -66,9 +65,11 @@ spec: replicaIndices: - 0 - 2 + - 4 - clusterName: demo-worker replicaIndices: - 1 + - 3 slice: projectNamespace: kubeslice-demo-distributed-mariadb sliceName: demo-slice @@ -82,26 +83,24 @@ spec: Here, -- `spec.clusterSpreadConstraint.distributionRules[].replicaIndices` specifies which MariaDB replica indices are scheduled on that cluster. Here `demo-controller` hosts replicas `0` and `2`, and `demo-worker` hosts replica `1`. +- `spec.clusterSpreadConstraint.distributionRules[].replicaIndices` specifies which MariaDB replica indices are scheduled on that cluster. `demo-controller` will host replicas `0`, `2`, and `4`; `demo-worker` will host replicas `1` and `3`. Apply the `PlacementPolicy` on the hub (`demo-controller`) cluster: ```bash -$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/autoscaler/compute/cluster/examples/placement-policy.yaml --context demo-controller +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml --context demo-controller placementpolicy.apps.k8s.appscode.com/distributed-mariadb created ``` ### Deploy Distributed MariaDB Cluster -In this section, we are going to deploy a distributed MariaDB Galera cluster with version `11.5.2`. Then, in the next section we will set up autoscaling for this database using `MariaDBAutoscaler` CRD. - -Below is the YAML of the `MariaDB` CR that we are going to create. Note that `spec.distributed` is set to `true` and the `PlacementPolicy` is referenced via `spec.podTemplate.spec.podPlacementPolicy`: +In this section, we are going to deploy a distributed MariaDB Galera cluster with version `11.5.2`. Note that `spec.distributed` is set to `true` and `spec.replicas` is `3` — which is less than the five indices defined in the `PlacementPolicy`. The operator will only create pods for indices `0`, `1`, and `2`. ```yaml apiVersion: kubedb.com/v1 kind: MariaDB metadata: - name: sample-mariadb + name: mariadb namespace: demo spec: version: "11.5.2" @@ -120,52 +119,73 @@ spec: podPlacementPolicy: name: distributed-mariadb containers: - - name: mariadb - resources: - requests: - cpu: "200m" - memory: "300Mi" - limits: - cpu: "200m" - memory: "300Mi" + - name: mariadb + resources: + requests: + cpu: "200m" + memory: "300Mi" + limits: + cpu: "200m" + memory: "300Mi" deletionPolicy: WipeOut ``` -Let's create the `MariaDB` CRO we have shown above, +Let's create the `MariaDB` CR we have shown above, ```bash -$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/autoscaler/compute/cluster/examples/sample-mariadb.yaml --context demo-controller -mariadb.kubedb.com/sample-mariadb created +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml --context demo-controller +mariadb.kubedb.com/mariadb created ``` -Now, wait until `sample-mariadb` has status `Ready`. i.e, +Now, wait until `mariadb` has status `Ready`: ```bash $ kubectl get mariadb -n demo --context demo-controller NAME VERSION STATUS AGE -sample-mariadb 11.5.2 Ready 14m +mariadb 11.5.2 Ready 2m36s ``` -The pods are distributed across clusters as defined by the `PlacementPolicy`: +Let's check the number of replicas this database has from the MariaDB object: ```bash -$ kubectl get pod -n demo --context demo-controller -NAME READY STATUS RESTARTS AGE -mariadb-0 3/3 Running 0 14m -mariadb-1 3/3 Running 0 14m +$ kubectl get mariadb -n demo mariadb -o json | jq '.spec.replicas' +3 +``` -$ kubectl get pod -n demo --context demo-worker -NAME READY STATUS RESTARTS AGE -mariadb-2 3/3 Running 0 15m +The pods are distributed across clusters as defined by the `PlacementPolicy`. Indices `0` and `2` land on `demo-controller`; index `1` lands on `demo-worker`: +```bash +$ kubectl get pods -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 2m30s +mariadb-2 3/3 Running 0 2m30s + +$ kubectl get pods -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-1 3/3 Running 0 2m30s +``` + +We can also verify the cluster size from inside a MariaDB pod: + +```bash +$ kubectl exec -it -n demo mariadb-0 -c mariadb --context demo-controller -- bash +root@mariadb-0:/ mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "show status like 'wsrep_cluster_size';" ++--------------------+-------+ +| Variable_name | Value | ++--------------------+-------+ +| wsrep_cluster_size | 3 | ++--------------------+-------+ +``` + +We can see the cluster has 3 nodes. We are now ready to apply `MariaDBOpsRequest` to scale this database. ## Scale Up Replicas -Here, we are going to scale up the replicas of the replicaset to meet the desired number of replicas after scaling. +Here, we are going to scale up the replicas from `3` to `5`. Because the `PlacementPolicy` already defines indices `0`–`4`, no changes to the policy are required — the operator knows where to place the new pods. -#### Create MariaDBOpsRequest +### Create MariaDBOpsRequest -In order to scale up the replicas of the replicaset of the database, we have to create a `MariaDBOpsRequest` CR with our desired replicas. Below is the YAML of the `MariaDBOpsRequest` CR that we are going to create, +In order to scale up the replicas, we have to create a `MariaDBOpsRequest` CR with our desired replicas. Below is the YAML of the `MariaDBOpsRequest` CR that we are going to create: ```yaml apiVersion: ops.kubedb.com/v1alpha1 @@ -183,36 +203,143 @@ spec: Here, -- `spec.databaseRef.name` specifies that we are performing horizontal scaling operation on `sample-mariadb` database. -- `spec.type` specifies that we are performing `HorizontalScaling` on our database. -- `spec.horizontalScaling.member` specifies the desired replicas after scaling. +- `spec.databaseRef.name` specifies that we are performing horizontal scaling on `mariadb`. +- `spec.type` specifies that we are performing `HorizontalScaling`. +- `spec.horizontalScaling.member` specifies the desired replica count after scaling. This value must not exceed the total number of indices defined in the `PlacementPolicy` (5 in this example). Let's create the `MariaDBOpsRequest` CR we have shown above, ```bash -$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/scaling/horizontal-scaling/cluster/example/mdops-upscale.yaml +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml --context demo-controller mariadbopsrequest.ops.kubedb.com/mdops-scale-horizontal-up created ``` -#### Verify Cluster replicas scaled up successfully +### Verify Cluster Replicas Scaled Up Successfully -If everything goes well, `KubeDB` Enterprise operator will update the replicas of `MariaDB` object and related `PetSets` and `Pods`. +If everything goes well, `KubeDB` Enterprise operator will update the replicas of the `MariaDB` object and create the new pods on the clusters specified by the `PlacementPolicy`. -Let's wait for `MariaDBOpsRequest` to be `Successful`. Run the following command to watch `MariaDBOpsRequest` CR, +Let's wait for `MariaDBOpsRequest` to be `Successful`: ```bash -$ watch kubectl get mariadbopsrequest -n demo +$ watch kubectl get mariadbopsrequest -n demo --context demo-controller Every 2.0s: kubectl get mariadbopsrequest -n demo NAME TYPE STATUS AGE -mdps-scale-horizontal HorizontalScaling Successful 106s +mdops-scale-horizontal-up HorizontalScaling Successful 18m ``` -We can see from the above output that the `MariaDBOpsRequest` has succeeded. Now, we are going to verify the number of replicas this database has from the MariaDB object, number of pods the petset have, +We can see from the above output that the `MariaDBOpsRequest` has succeeded. Now, let's verify the number of replicas: ```bash -$ kubectl get mariadb -n demo sample-mariadb -o json | jq '.spec.replicas' -5 -$ kubectl get sts -n demo sample-mariadb -o json | jq '.spec.replicas' +$ kubectl get mariadb -n demo mariadb -o json | jq '.spec.replicas' 5 ``` +The two new pods are placed on the clusters according to the `PlacementPolicy` — index `4` on `demo-controller` and index `3` on `demo-worker`: + +```bash +$ kubectl get pods -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 58m +mariadb-2 3/3 Running 0 55m +mariadb-4 3/3 Running 0 17m + + +$ kubectl get pods -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-1 3/3 Running 0 57m +mariadb-3 3/3 Running 0 19m +``` + +From all the above outputs we can see that the cluster now has `5` replicas. We have successfully scaled up the distributed MariaDB cluster. + +## Scale Down Replicas + +Here, we are going to scale down the replicas from `5` to `3`. The `PlacementPolicy` does not need to be modified — the operator will remove the highest-indexed pods (`mariadb-3` and `mariadb-4`) and the remaining pods will continue running as defined by the policy. + +### Create MariaDBOpsRequest + +In order to scale down the cluster, we have to create a `MariaDBOpsRequest` CR with our desired replicas. Below is the YAML of the `MariaDBOpsRequest` CR that we are going to create: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member: 3 +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing horizontal scaling down on `mariadb`. +- `spec.type` specifies that we are performing `HorizontalScaling`. +- `spec.horizontalScaling.member` specifies the desired replica count after scaling. + +Let's create the `MariaDBOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml --context demo-controller +mariadbopsrequest.ops.kubedb.com/mdops-scale-horizontal-down created +``` + +### Verify Cluster Replicas Scaled Down Successfully + +If everything goes well, `KubeDB` Enterprise operator will update the replicas of the `MariaDB` object and remove the highest-indexed pods from their respective clusters. + +Let's wait for `MariaDBOpsRequest` to be `Successful`: + +```bash +$ watch kubectl get mariadbopsrequest -n demo --context demo-controller +Every 2.0s: kubectl get mariadbopsrequest -n demo +NAME TYPE STATUS AGE +mdops-scale-horizontal-down HorizontalScaling Successful 2m32s +``` + +We can see from the above output that the `MariaDBOpsRequest` has succeeded. Now, let's verify the number of replicas: + +```bash +$ kubectl get mariadb -n demo mariadb -o json | jq '.spec.replicas' +3 +``` + +Pods `mariadb-3` and `mariadb-4` have been removed from their respective clusters: + +```bash +$ kubectl get pods -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 20m +mariadb-2 3/3 Running 0 20m + +$ kubectl get pods -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-1 3/3 Running 0 20m +``` + +Let's verify the cluster size from inside a MariaDB pod: + +```bash +$ kubectl exec -it -n demo mariadb-0 -c mariadb --context demo-controller -- bash +root@mariadb-0:/ mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "show status like 'wsrep_cluster_size';" ++--------------------+-------+ +| Variable_name | Value | ++--------------------+-------+ +| wsrep_cluster_size | 3 | ++--------------------+-------+ +``` + +From all the above outputs we can see that the cluster now has `3` replicas. We have successfully scaled down the distributed MariaDB cluster. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +$ kubectl delete mariadb -n demo mariadb --context demo-controller +$ kubectl delete mariadbopsrequest -n demo mdops-scale-horizontal-up mdops-scale-horizontal-down --context demo-controller +$ kubectl delete placementpolicy distributed-mariadb --context demo-controller +``` diff --git a/docs/guides/postgres/distributed/overview/index.md b/docs/guides/postgres/distributed/overview/index.md index 3775d2970d..458e77a33b 100644 --- a/docs/guides/postgres/distributed/overview/index.md +++ b/docs/guides/postgres/distributed/overview/index.md @@ -9,11 +9,12 @@ menu: menu_name: docs_{{ .version }} section_menu_id: guides --- + # Distributed Postgres Cluster Overview ## Introduction -KubeDB enables distributed Postgres deployments across multiple Kubernetes clusters, providing a scalable, highly available, and resilient database solution. By integrating **Open Cluster Management (OCM)** for multi-cluster orchestration and **KubeSlice** for seamless pod-to-pod network connectivity, KubeDB simplifies the deployment and management of Postgres instances across clusters. A **PodPlacementPolicy** ensures precise control over pod scheduling, allowing you to distribute Postgres pods across clusters for optimal resource utilization and fault tolerance. +KubeDB enables distributed Postgres deployments across multiple Kubernetes clusters, providing a scalable, highly available, and resilient database solution. By integrating **Open Cluster Management (OCM)** for multi-cluster orchestration and **KubeSlice** for seamless pod-to-pod network connectivity, KubeDB simplifies the deployment and management of Postgres instances across clusters. A **PlacementPolicy** ensures precise control over pod scheduling, allowing you to distribute Postgres pods across clusters for optimal resource utilization and fault tolerance. This guide provides a step-by-step process to deploy a distributed Postgres cluster, including prerequisites, configuration, and verification steps. It assumes familiarity with Kubernetes and basic database concepts. @@ -21,23 +22,23 @@ This guide provides a step-by-step process to deploy a distributed Postgres clus ## Understanding OCM Hub and Spoke Clusters -In an **Open Cluster Management (OCM)** setup, clusters are categorized as: +In an **Open Cluster Management (OCM)** setup, clusters are categorized as follows: -* **Hub Cluster**: The central control plane where policies, [applications](), and resources are defined and managed. It orchestrates the lifecycle of applications deployed across spoke clusters. -* **Spoke Cluster**: Managed clusters registered with the hub, running the actual workloads (e.g., Postgres pods). +- **Hub Cluster**: The central control plane where policies, applications, and resources are defined and managed. It orchestrates the lifecycle of applications deployed across spoke clusters. +- **Spoke Cluster**: Managed clusters registered with the hub that run the actual workloads (e.g., Postgres pods). -When a spoke cluster (e.g., `demo-worker`) is joined to the hub using the `clusteradm join` command, OCM creates a namespace on the hub cluster matching the spoke cluster's name (e.g., `demo-worker`). This namespace is used to manage resources specific to the spoke cluster from the hub. +When a spoke cluster (e.g., `demo-worker`) is joined to the hub using the `clusteradm join` command, OCM creates a namespace on the hub cluster that matches the spoke cluster's name (e.g., `demo-worker`). This namespace is used to manage resources specific to the spoke cluster from the hub. ## Prerequisites -Before deploying a distributed Postgres cluster, ensure the following: +Before deploying a distributed Postgres cluster, ensure the following requirements are met: - **Kubernetes Clusters**: Multiple Kubernetes clusters (version 1.26 or higher) configured and accessible. -- **Node Requirements**: Each Kubernetes node should have at least 4 vCPUs and 16GB of RAM. -- **Open Cluster Management (OCM)**: Install `clusteradm` as per the [OCM Quick Start Guide](https://open-cluster-management.io/docs/getting-started/quick-start/). +- **Node Requirements**: Each Kubernetes node should have at least 4 vCPUs and 16 GB of RAM. +- **Open Cluster Management (OCM)**: Install `clusteradm` as described in the [OCM Quick Start Guide](https://open-cluster-management.io/docs/getting-started/quick-start/). - **kubectl**: Installed and configured to interact with all clusters. - **Helm**: Installed for deploying the KubeDB Operator and KubeSlice components. -- **Persistent Storage**: A storage class (e.g., `local-path` or cloud provider-specific) configured for persistent volumes. +- **Persistent Storage**: A storage class (e.g., `local-path` or a cloud provider-specific option) configured for persistent volumes. ## Configuration Steps @@ -45,164 +46,197 @@ Follow these steps to deploy a distributed Postgres cluster across multiple Kube ### Step 1: Set Up Open Cluster Management (OCM) -1. **Configure KUBECONFIG**: - Ensure your `KUBECONFIG` is set up to switch between clusters. This guide uses two clusters: `demo-controller` (hub and spoke) and `demo-worker` (spoke). - - ```bash - kubectl config get-contexts - ``` - - **Output**: - ``` - CURRENT NAME CLUSTER AUTHINFO NAMESPACE - * demo-controller demo-controller demo-controller - demo-worker demo-worker demo-worker - ``` - -2. **Initialize the OCM Hub**: - On the `demo-controller` cluster, initialize the OCM hub. - - ```bash - kubectl config use-context demo-controller - clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true - ``` - -3. **Verify Hub Deployment**: - Check the pods in the `open-cluster-management-hub` namespace to ensure all components are running. - - ```bash - kubectl get pods -n open-cluster-management-hub - ``` - - **Output**: - ``` - NAME READY STATUS RESTARTS AGE - cluster-manager-addon-manager-controller-5f99f56896-qpzj8 1/1 Running 0 7m2s - cluster-manager-placement-controller-597d5ff644-wqjq2 1/1 Running 0 7m2s - cluster-manager-registration-controller-6d79d7dcc6-b8h9p 1/1 Running 0 7m2s - cluster-manager-registration-webhook-5d88cf97c7-2sq5m 1/1 Running 0 7m2s - cluster-manager-work-controller-7468bf4dc-5qn6q 1/1 Running 0 7m2s - cluster-manager-work-webhook-c5875947-d272b 1/1 Running 0 7m2s - ``` - - All pods should be in the `Running` state with `1/1` readiness and no restarts, indicating a successful hub deployment. - -4. **Register Spoke Cluster (`demo-worker`)**: - Obtain the join token from the hub cluster. - - ```bash - clusteradm get token - ``` - - **Output**: - ``` - token= - please log on spoke and run: - clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name - ``` - - On the `demo-worker` cluster, join it to the hub, including the `RawFeedbackJsonString` feature gate for resource feedback. - - ```bash - kubectl config use-context demo-worker - clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true - ``` - -5. **Accept Spoke Cluster**: - On the `demo-controller` cluster, accept the `demo-worker` cluster. - - ```bash - kubectl config use-context demo-controller - clusteradm accept --clusters demo-worker - ``` - - **Note**: It may take a few attempts (e.g., retry every 10 seconds) if the cluster is not immediately available. - - **Output** (on success): - ``` - Starting approve csrs for the cluster demo-worker - CSR demo-worker-2p2pb approved - set hubAcceptsClient to true for managed cluster demo-worker - Your managed cluster demo-worker has joined the Hub successfully. - ``` - -6. **Verify Namespace Creation**: - Confirm that a namespace for `demo-worker` was created on the hub cluster. - - ```bash - kubectl get ns - ``` - - **Output**: - ``` - NAME STATUS AGE - default Active 99m - demo-worker Active 58s - kube-node-lease Active 99m - kube-public Active 99m - kube-system Active 99m - open-cluster-management Active 6m7s - open-cluster-management-hub Active 5m32s - ``` - -7. **Register `demo-controller` as a Spoke Cluster**: - Repeat the join and accept process for `demo-controller` to also act as a spoke cluster. - - ```bash - kubectl config use-context demo-controller - clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true - clusteradm accept --clusters demo-controller - ``` - - Verify the namespace for `demo-controller`. - - ```bash - kubectl get ns - ``` - - **Output**: - ``` - NAME STATUS AGE - default Active 104m - demo-controller Active 3s - demo-worker Active 6m7s - kube-node-lease Active 104m - kube-public Active 104m - kube-system Active 104m - open-cluster-management Active 11m - open-cluster-management-agent Active 37s - open-cluster-management-agent-addon Active 34s - open-cluster-management-hub Active 10m - ``` - -### Step 2: Configure OCM WorkConfiguration (Optional) - -If you did not follow the provided OCM installation steps, update the `klusterlet` resource to enable feedback retrieval. - -```bash -kubectl edit klusterlet klusterlet -``` +#### 1. Configure KUBECONFIG -Add the following under the `spec` field: +Ensure your `KUBECONFIG` is set up to switch between clusters. This guide uses two clusters: `demo-controller` (hub and spoke) and `demo-worker` (spoke). + +```bash +$ kubectl config get-contexts +``` + +**Output:** + +```bash +CURRENT NAME CLUSTER AUTHINFO NAMESPACE +* demo-controller demo-controller demo-controller + demo-worker demo-worker demo-worker +``` + +#### 2. Initialize the OCM Hub + +On the `demo-controller` cluster, initialize the OCM hub: + +```bash +$ kubectl config use-context demo-controller +$ clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true +``` + +#### 3. Verify Hub Deployment + +Check the pods in the `open-cluster-management-hub` namespace to ensure all components are running: + +```bash +$ kubectl get pods -n open-cluster-management-hub +``` + +**Output:** -```yaml -workConfiguration: - featureGates: - - feature: RawFeedbackJsonString - mode: Enable - hubKubeAPIBurst: 100 - hubKubeAPIQPS: 50 - kubeAPIBurst: 100 - kubeAPIQPS: 50 +``` +NAME READY STATUS RESTARTS AGE +cluster-manager-addon-manager-controller-5f99f56896-qpzj8 1/1 Running 0 7m2s +cluster-manager-placement-controller-597d5ff644-wqjq2 1/1 Running 0 7m2s +cluster-manager-registration-controller-6d79d7dcc6-b8h9p 1/1 Running 0 7m2s +cluster-manager-registration-webhook-5d88cf97c7-2sq5m 1/1 Running 0 7m2s +cluster-manager-work-controller-7468bf4dc-5qn6q 1/1 Running 0 7m2s +cluster-manager-work-webhook-c5875947-d272b 1/1 Running 0 7m2s +``` + +All pods should be in the `Running` state with `1/1` readiness and no restarts, indicating a successful hub deployment. + +#### 4. Register Spoke Cluster (`demo-worker`) + +Obtain the join token from the hub cluster: + +```bash +$ clusteradm get token +``` + +**Output:** + +``` +token= +please log on spoke and run: +clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name +``` + +On the `demo-worker` cluster, join it to the hub. Include the `RawFeedbackJsonString` feature gate for resource feedback: + +```bash +$ kubectl config use-context demo-worker +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true +``` + +#### 5. Accept Spoke Cluster + +On the `demo-controller` cluster, accept the `demo-worker` cluster: + +```bash +$ kubectl config use-context demo-controller +$ clusteradm accept --clusters demo-worker +``` + +> **Note:** It may take a few attempts (e.g., retry every 10 seconds) if the cluster is not immediately available. + +**Output (on success):** + + +- Starting approve csrs for the cluster demo-worker +- CSR demo-worker-2p2pb approved +- set hubAcceptsClient to true for managed cluster demo-worker +- Your managed cluster demo-worker has joined the Hub successfully. + + +#### 6. Verify Namespace Creation + +Confirm that a namespace for `demo-worker` was created on the hub cluster: + +```bash +$ kubectl get ns +``` + +**Output:** + +```bash +NAME STATUS AGE +default Active 99m +demo-worker Active 58s +kube-node-lease Active 99m +kube-public Active 99m +kube-system Active 99m +open-cluster-management Active 6m7s +open-cluster-management-hub Active 5m32s +``` + +#### 7. Register `demo-controller` as a Spoke Cluster + +Repeat the join and accept process for `demo-controller` so it can also act as a spoke cluster: + +```bash +$ kubectl config use-context demo-controller +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true +$ clusteradm accept --clusters demo-controller +``` + +Verify the namespace for `demo-controller`: + +```bash +$ kubectl get ns +``` + +**Output:** + +``` +NAME STATUS AGE +default Active 104m +demo-controller Active 3s +demo-worker Active 6m7s +kube-node-lease Active 104m +kube-public Active 104m +kube-system Active 104m +open-cluster-management Active 11m +open-cluster-management-agent Active 37s +open-cluster-management-agent-addon Active 34s +open-cluster-management-hub Active 10m +``` + +#### 8. Verify OCM Roles + +After registration, use these commands to confirm which cluster is the hub and which are spokes: + +```bash +# Hub: lists all registered spoke clusters +$ kubectl get managedclusters + +# Spoke: shows this cluster's registered name +$ kubectl get klusterlet klusterlet -o jsonpath='{.spec.clusterName}' + +# Hub components run only on the hub cluster +$ kubectl get pods -n open-cluster-management-hub + +# Spoke agent runs on every spoke cluster +$ kubectl get pods -n open-cluster-management-agent +``` + +### Step 2: Configure OCM WorkConfiguration + +Run this on **every spoke cluster** (`demo-controller` and `demo-worker`). This enables the `RawFeedbackJsonString` feature gate that KubeDB requires to read pod status across clusters, and raises the API rate limits to prevent throttling. + +> **Note:** Even if you passed `--feature-gates=RawFeedbackJsonString=true` during `clusteradm join`, the rate limit fields are not set by that flag. Run this patch on all spokes regardless. +> +> **Why this matters:** KubeDB uses OCM's ManifestWork feedback mechanism to watch the status of Postgres pods on remote spoke clusters. Without `RawFeedbackJsonString`, the KubeDB provisioner on the hub never receives pod status updates from spokes and the distributed Postgres CR will stay in a non-Ready state indefinitely. The rate limits prevent the klusterlet agent from being API-throttled during initial cluster formation. + +```bash +$ kubectl patch klusterlet klusterlet --type=merge -p '{ + "spec": { + "workConfiguration": { + "featureGates": [{"feature": "RawFeedbackJsonString", "mode": "Enable"}], + "hubKubeAPIBurst": 100, + "hubKubeAPIQPS": 50, + "kubeAPIBurst": 100, + "kubeAPIQPS": 50 + } + } +}' ``` Verify the configuration: ```bash -kubectl get klusterlet klusterlet -oyaml +$ kubectl get klusterlet klusterlet -oyaml ``` -**Sample Output** (abridged): +**Sample Output (abridged):** + ```yaml apiVersion: operator.open-cluster-management.io/v1 kind: Klusterlet @@ -220,290 +254,235 @@ spec: kubeAPIQPS: 50 ``` - ### Step 3: Configure KubeSlice for Network Connectivity KubeSlice enables pod-to-pod communication across clusters. Install the KubeSlice Controller on the `demo-controller` cluster and the KubeSlice Worker on both `demo-controller` and `demo-worker` clusters. -1. **Install KubeSlice Controller**: - On `demo-controller`, create a `controller.yaml` file: - - ```yaml - kubeslice: - controller: - loglevel: info - rbacResourcePrefix: kubeslice-rbac - projectnsPrefix: kubeslice - endpoint: https://10.2.0.56:6443 - ``` - - Deploy the controller using Helm: - - ```bash - helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ - --version v2025.7.31 \ - -f controller.yaml \ - --namespace kubeslice-controller \ - --create-namespace \ - --wait --burst-limit=10000 --debug - ``` - - Verify the installation: - - ```bash - kubectl get pods -n kubeslice-controller - ``` - - **Output**: - ``` - NAME READY STATUS RESTARTS AGE - kubeslice-controller-manager-7fd756fff6-5kddd 2/2 Running 0 98s - ``` - -2. **Create a KubeSlice Project**: - Create a `project.yaml` file: - - ```yaml - apiVersion: controller.kubeslice.io/v1alpha1 - kind: Project - metadata: - name: demo-distributed-postgres - namespace: kubeslice-controller - spec: - serviceAccount: - readWrite: - - admin - ``` - - Apply the project: - - ```bash - kubectl apply -f project.yaml - ``` - - Verify: - - ```bash - kubectl get project -n kubeslice-controller - ``` - - **Output**: - ``` - NAME AGE - demo-distributed-postgres 31s - ``` - - Check service accounts: - - ```bash - kubectl get sa -n kubeslice-demo-distributed-postgres - ``` - - **Output**: - ``` - NAME SECRETS AGE - default 0 69s - kubeslice-rbac-rw-admin 1 68s - ``` - -3. **Label Nodes for KubeSlice**: - Assign the `kubeslice.io/node-type=gateway` label to node(where worker operator will deploy) in both clusters. - - On `demo-controller`: - - ```bash - kubectl get nodes - kubectl label node demo-master kubeslice.io/node-type=gateway - ``` - - On `demo-worker`: - - ```bash - kubectl config use-context demo-worker - kubectl get nodes - kubectl label node demo-worker kubeslice.io/node-type=gateway - ``` - -4. **Register Clusters with KubeSlice**: - Identify the network interface for both clusters by running the command on the node. - - ```bash - ip route get 8.8.8.8 | awk '{ print $5 }' - ``` - - **Output** (example): - ``` - enp1s0 - ``` - - Create a `registration.yaml` file: - - ```yaml - apiVersion: controller.kubeslice.io/v1alpha1 - kind: Cluster - metadata: - name: demo-controller - namespace: kubeslice-demo-distributed-postgres - spec: - networkInterface: enp1s0 - clusterProperty: {} - --- - apiVersion: controller.kubeslice.io/v1alpha1 - kind: Cluster - metadata: - name: demo-worker - namespace: kubeslice-demo-distributed-postgres - spec: - networkInterface: enp1s0 - clusterProperty: {} - ``` - - Apply on `demo-controller`: - - ```bash - kubectl apply -f registration.yaml - ``` - - Verify: - - ```bash - kubectl get clusters -n kubeslice-demo-distributed-postgres - ``` - - **Output**: - ``` - NAME AGE - demo-controller 9s - demo-worker 9s - ``` - -5. **Register KubeSlice Worker Clusters**: - Create a `secrets.sh` script to generate worker configuration: - -```bash - # The script returns a kubeconfig for the service account given -# you need to have kubectl on PATH with the context set to the cluster you want to create the config for -# Cosmetics for the created config -firstWorkerSecretName=$1 -# cluster name what you given in clusters registration -clusterName=$2 -# the Namespace and ServiceAccount name that is used for the config -namespace=$3 -# Need to give correct network interface value like ens160, eth0 etc -networkInterface=$4 -# kubectl cluster-info of respective worker-cluster -worker_endpoint=$5 -###################### -# actual script starts -set -o errexit - -### Fetch Worker cluster Secrets ### -PROJECT_NAMESPACE=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath={.data.namespace}) -CONTROLLER_ENDPOINT=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath={.data.controllerEndpoint}) -CA_CRT=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath='{.data.ca\.crt}') -TOKEN=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath={.data.token}) - -echo " +#### 1. Install KubeSlice Controller + +On `demo-controller`, get the hub API server address first: + +```bash +$ kubectl cluster-info | grep 'Kubernetes control plane' +``` + +Use the IP and port from that output as the `endpoint` value. Create a `controller.yaml` file: + +```yaml +kubeslice: + controller: + loglevel: info + rbacResourcePrefix: kubeslice-rbac + projectnsPrefix: kubeslice + endpoint: https://:6443 +``` + +Deploy the controller using Helm: + +```bash +$ helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ + --version v2026.1.15 \ + -f controller.yaml \ + --namespace kubeslice-controller \ + --create-namespace \ + --set ocm.enabled=true \ + --wait --burst-limit=10000 --debug +``` + +Verify the installation: + +```bash +$ kubectl get pods -n kubeslice-controller +``` + +**Output:** + +``` +NAME READY STATUS RESTARTS AGE +kubeslice-controller-manager-7fd756fff6-5kddd 2/2 Running 0 98s +``` + +#### 2. Create a KubeSlice Project + +Create a `project.yaml` file: + +```yaml +apiVersion: controller.kubeslice.io/v1alpha1 +kind: Project +metadata: + name: demo-distributed-postgres + namespace: kubeslice-controller +spec: + serviceAccount: + readWrite: + - admin +``` + +Apply the project: + +```bash +$ kubectl apply -f project.yaml +``` + +Verify: + +```bash +$ kubectl get project -n kubeslice-controller +``` + +**Output:** + +```bash +NAME AGE +demo-distributed-postgres 31s +``` + +Check service accounts: + +```bash +$ kubectl get sa -n kubeslice-demo-distributed-postgres +``` + +**Output:** + +``` +NAME SECRETS AGE +default 0 69s +kubeslice-rbac-rw-admin 1 68s +``` + +#### 3. Label Nodes for KubeSlice + +Assign the `kubeslice.io/node-type=gateway` label to the node where the worker operator will be deployed in both clusters. + +On `demo-controller`: + +```bash +$ kubectl get nodes +$ kubectl label node kubeslice.io/node-type=gateway +``` + +On `demo-worker`: + +```bash +$ kubectl config use-context demo-worker +$ kubectl get nodes +$ kubectl label node kubeslice.io/node-type=gateway +``` + +#### 4. Register Clusters with KubeSlice + +Identify the network interface for each cluster by running the following command **on the gateway node of each cluster**: + +```bash +$ ip route get 8.8.8.8 | awk '{ print $5 }' +``` + +**Output (example):** + +``` +enp1s0 +``` + +> **Important:** The `networkInterface` value must match the primary network interface of each cluster's gateway node. Run the command above on **each cluster separately** and use that cluster's output as its `networkInterface` value. Each cluster may have a **different interface name** — do not assume they are the same. Using the wrong interface name will cause the WireGuard gateway to silently fail and cross-cluster Postgres replication will never connect. +> +> Example: if `demo-controller` returns `enp3s0` and `demo-worker` returns `eth0`, use those exact values in the YAML below. + +Create a `registration.yaml` file: + +> **Note:** +> - The cluster name must exactly match the name of the OCM (spoke) cluster. +> - The corresponding `ManagedClusterAddOn` resource must be created in the namespace that bears the same name as the cluster to set up the KubeSlice worker automatically. + +```yaml +apiVersion: controller.kubeslice.io/v1alpha1 +kind: Cluster +metadata: + name: demo-controller + namespace: kubeslice-demo-distributed-postgres +spec: + networkInterface: # replace with output of ip route command + clusterProperty: {} +--- +apiVersion: controller.kubeslice.io/v1alpha1 +kind: Cluster +metadata: + name: demo-worker + namespace: kubeslice-demo-distributed-postgres +spec: + networkInterface: # replace with output of ip route command + clusterProperty: {} --- -## Base64 encoded secret values from controller cluster -controllerSecret: -namespace: ${PROJECT_NAMESPACE} -endpoint: ${CONTROLLER_ENDPOINT} -ca.crt: ${CA_CRT} -token: ${TOKEN} -cluster: -name: ${clusterName} -endpoint: ${worker_endpoint} -netop: -networkInterface: ${networkInterface} -" -``` -Command to generate the worker configuration: -```bash -sh secrets.sh -``` - -Get secrets for the project namespace: - - ```bash - kubectl get secrets -n kubeslice-demo-distributed-postgres - ``` - -**Output**: - ``` - NAME TYPE DATA AGE - kubeslice-rbac-rw-admin kubernetes.io/service-account-token 3 17m - kubeslice-rbac-worker-demo-controller kubernetes.io/service-account-token 5 5m8s - kubeslice-rbac-worker-demo-worker kubernetes.io/service-account-token 5 5m8s - ``` - -Get the `demo-worker` cluster endpoint: - - ```bash - kubectl cluster-info --context demo-worker --kubeconfig $HOME/.kube/config - ``` - -**Output**: - ``` - Kubernetes control plane is running at https://10.2.0.60:6443 - ``` - -Run the script for `demo-worker`: - - ```bash - sh secrets.sh kubeslice-rbac-worker-demo-worker demo-worker kubeslice-demo-distributed-postgres enp1s0 https://10.2.0.60:6443 > sliceoperator-worker.yaml - ``` - -Install the KubeSlice worker on `demo-worker`: - - ```bash - kubectl config use-context demo-worker - helm upgrade -i kubeslice-worker oci://ghcr.io/appscode-charts/kubeslice-worker \ - --version v2025.7.31 \ - -f sliceoperator-worker.yaml \ - --namespace kubeslice-system \ - --create-namespace \ - --wait --burst-limit=10000 --debug - ``` - -Repeat for `demo-controller`: - - ```bash - kubectl config use-context demo-controller - sh secrets.sh kubeslice-rbac-worker-demo-controller demo-controller kubeslice-demo-distributed-postgres enp1s0 https://10.2.0.56:6443 > sliceoperator-controller.yaml - helm upgrade -i kubeslice-worker oci://ghcr.io/appscode-charts/kubeslice-worker \ - --version v2025.7.31 \ - -f sliceoperator-controller.yaml \ - --namespace kubeslice-system \ - --create-namespace \ - --wait --burst-limit=10000 --debug - ``` - -Verify the worker installation: - - ```bash - kubectl get pods -n kubeslice-system - ``` - -**Output**: - ``` - NAME READY STATUS RESTARTS AGE - forwarder-kernel-bw5l4 1/1 Running 0 4m43s - kubeslice-dns-6bd9749f4d-pvh7g 1/1 Running 0 4m43s - kubeslice-install-crds-szhvc 0/1 Completed 0 4m56s - kubeslice-netop-g4dfn 1/1 Running 0 4m43s - kubeslice-operator-949b7d6f7-9wj7h 2/2 Running 0 4m43s - kubeslice-postdelete-job-ctlzt 0/1 Completed 0 20m - nsm-delete-webhooks-ndksl 0/1 Completed 0 20m - nsm-install-crds-5z4j9 0/1 Completed 0 4m53s - nsmgr-zzwgh 2/2 Running 0 4m43s - registry-k8s-979455d6d-q2j8x 1/1 Running 0 4m43s - spire-install-clusterid-cr-qwqlr 0/1 Completed 0 4m47s - spire-install-crds-cnbjh 0/1 Completed 0 4m50s - ``` - - **Onboard Database Namespace** +apiVersion: addon.open-cluster-management.io/v1alpha1 +kind: ManagedClusterAddOn +metadata: + name: kubeslice + namespace: demo-controller +spec: + installNamespace: kubeslice-system + configs: + - name: demo-controller + namespace: kubeslice-demo-distributed-postgres + group: controller.kubeslice.io + resource: clusters +--- +apiVersion: addon.open-cluster-management.io/v1alpha1 +kind: ManagedClusterAddOn +metadata: + name: kubeslice + namespace: demo-worker +spec: + installNamespace: kubeslice-system + configs: + - name: demo-worker + namespace: kubeslice-demo-distributed-postgres + group: controller.kubeslice.io + resource: clusters +``` +Apply on `demo-controller`: + +```bash +$ kubectl apply -f registration.yaml +``` + +Verify OCM is deploying the KubeSlice worker manifests to each cluster: + +```bash +$ kubectl get managedclusteraddon -A +``` + +**Output:** + +``` +NAMESPACE NAME AVAILABLE DEGRADED PROGRESSING +demo-controller kubeslice Unknown True +demo-worker kubeslice Unknown True +``` + +`PROGRESSING: True` means OCM is actively deploying. Wait until `kubeslice-operator` shows `2/2 Running` on both clusters before proceeding: + +```bash +# Run on each spoke cluster +$ kubectl get pods -n kubeslice-system --watch +``` + +**Expected output (after KubeSlice worker is fully deployed):** + +``` +NAME READY STATUS RESTARTS AGE +forwarder-kernel-bw5l4 1/1 Running 0 4m43s +kubeslice-dns-6bd9749f4d-pvh7g 1/1 Running 0 4m43s +kubeslice-install-crds-szhvc 0/1 Completed 0 4m56s +kubeslice-netop-g4dfn 1/1 Running 0 4m43s +kubeslice-operator-949b7d6f7-9wj7h 2/2 Running 0 4m43s +nsc-grpc-server-sbjj7 1/1 Running 0 4m43s +nsm-install-crds-5z4j9 0/1 Completed 0 4m53s +nsmgr-zzwgh 2/2 Running 0 4m43s +registry-k8s-979455d6d-q2j8x 1/1 Running 0 4m43s +``` + +#### 5. Onboard Application Namespace Create a `SliceConfig` to onboard the `demo` (application) and `kubedb` (operator) namespaces for network connectivity. Create a `sliceconfig.yaml` file: @@ -552,75 +531,200 @@ spec: Apply the `SliceConfig`: ```bash -kubectl apply -f sliceconfig.yaml +$ kubectl apply -f sliceconfig.yaml +``` + +After the SliceConfig is applied, a `vl3-slice-router` pod will appear in `kubeslice-system` on each cluster, indicating the slice VPN tunnel is being established. + +#### 6. Configure DNS for KubeSlice + +Update CoreDNS to forward `*.slice.local` traffic to the KubeSlice DNS service. Run the following steps on **every cluster** in the slice. + +> **Important:** CoreDNS must be updated and restarted on all clusters before proceeding to Step 4 (KubeDB install). Postgres nodes use `.slice.local` DNS names to discover each other across clusters. If DNS is not configured before Postgres pods start, replication will not form. + +Get the KubeSlice DNS service IP address on each cluster: + +```bash +$ kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' ``` +**Output:** + +```bash +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR +kubeslice-dns ClusterIP 10.43.172.191 53/UDP,53/TCP 8d app=kubeslice-dns +``` + +Add the following block to the **top** of your CoreDNS `Corefile` ConfigMap (replace the IP with the one from your cluster): + +``` +slice.local:53 { + errors + cache 30 + forward . 10.43.172.191 +} +``` + +Example of the full CoreDNS ConfigMap after editing: + +```bash +$ kubectl get cm -n kube-system coredns -oyaml +``` + +**Output:** + +```yaml +apiVersion: v1 +data: + Corefile: | + slice.local:53 { + errors + cache 30 + forward . 10.43.172.191 + } + .:53 { + errors + health + ready + kubernetes cluster.local in-addr.arpa ip6.arpa { + pods insecure + fallthrough in-addr.arpa ip6.arpa + } + hosts /etc/coredns/NodeHosts { + ttl 60 + reload 15s + fallthrough + } + prometheus :9153 + cache 30 + loop + reload + loadbalance + import /etc/coredns/custom/*.override + forward . /etc/resolv.conf + } + import /etc/coredns/custom/*.server + NodeHosts: | + 10.2.0.248 demo-worker +kind: ConfigMap +metadata: + name: coredns + namespace: kube-system +``` + +After editing the ConfigMap, restart CoreDNS to apply the change: + +```bash +$ kubectl rollout restart deploy/coredns -n kube-system +``` + +Repeat the DNS configuration steps on every cluster in the slice. + ### Step 4: Install the KubeDB Operator -Install the KubeDB Operator on the `demo-controller` cluster to manage the Postgres instance. +> **Note:** Install the KubeDB operator only on the hub cluster (`demo-controller`). The operator manages Postgres pods on spoke clusters through OCM — no KubeDB installation is needed on `demo-worker`. #### Get a Free License -Download a FREE license from [AppsCode License Server](https://appscode.com/issue-license?p=kubedb). Get the license for `demo-controller` cluster. + +The KubeDB license is tied to the `kube-system` namespace UID of the hub cluster and has an expiry date. Get your cluster UID and verify the license before installing: ```bash -helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ - --version v2025.8.31 \ +# Get your cluster UID (required when requesting the license) +$ kubectl get ns kube-system -o jsonpath='{.metadata.uid}' + +# Verify the license is not expired +$ openssl x509 -noout -enddate -in $HOME/Downloads/kubedb-license-.txt +``` + +If expired or not yet obtained, download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb) using the cluster UID above. + +```bash +$ helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ + --version v2026.2.26 \ --namespace kubedb --create-namespace \ - --set-file global.license=$HOME/Downloads/kubedb-license-cd548cce-5141-4ed3-9276-6d9578707f12.txt \ + --set-file global.license=$HOME/Downloads/kubedb-license-.txt \ --set petset.features.ocm.enabled=true \ --wait --burst-limit=10000 --debug ``` -Note: `--set petset.features.ocm.enabled=true` must be set to enable Postgres Distributed feature. Also `--set-file global.license=` should point to your kubedb license file. -Follow the [KubeDB Installation Guide](https://kubedb.com/docs/v2025.6.30/setup/install/kubedb/) for additional details. +> **Note:** The `--set petset.features.ocm.enabled=true` flag must be set to enable the Postgres Distributed feature. + +For additional details, refer to the [KubeDB Installation Guide](https://kubedb.com/docs/v2026.4.27/setup/). + +Verify that the pods are running: + +```bash +$ kubectl get pods -n kubedb +``` + +**Output:** -### Step 5: Define a PodPlacementPolicy +```bash +NAME READY STATUS RESTARTS AGE +kubedb-kubedb-autoscaler-0 1/2 Running 0 44s +kubedb-kubedb-ops-manager-0 1/2 Running 0 43s +kubedb-kubedb-provisioner-0 1/2 Running 0 43s +kubedb-kubedb-webhook-server-df667cd85-tjdp9 2/2 Running 0 44s +kubedb-petset-cf9f5b6f4-d9558 2/2 Running 0 44s +kubedb-sidekick-5dbf7bcf64-4b8cw 2/2 Running 0 44s +``` -Create a `PlacementPolicy` to control pod distribution across clusters. Example: +### Step 5: Define a PlacementPolicy + +You can define the `storageClassName` under `spec.clusterSpreadConstraint.distributionRules` for each cluster. If not explicitly specified, the clusters will automatically select an appropriate storage class. Additionally, you can control replica placement by defining which replicas belong to which cluster. When scaling the number of replicas, the distribution must also be specified at the cluster level. In this example, we are using three replicas. + +To manage pod distribution across clusters, create a `PlacementPolicy`. For this purpose, define a `pod-placement-policy.yaml` file as shown below: ```yaml apiVersion: apps.k8s.appscode.com/v1 kind: PlacementPolicy metadata: - labels: - app.kubernetes.io/managed-by: Helm - name: distributed-postgres + labels: + app.kubernetes.io/managed-by: Helm + name: distributed-postgres spec: - clusterSpreadConstraint: - distributionRules: - - clusterName: demo-controller - replicaIndices: - - 0 - - 2 - - clusterName: demo-worker - replicaIndices: - - 1 - slice: - projectNamespace: kubeslice-demo-distributed-mariadb - sliceName: demo-slice - nodeSpreadConstraint: - maxSkew: 1 - whenUnsatisfiable: ScheduleAnyway - zoneSpreadConstraint: - maxSkew: 1 - whenUnsatisfiable: ScheduleAnyway - - + clusterSpreadConstraint: + distributionRules: + - clusterName: demo-controller + storageClassName: local-path # optional; omit to use the cluster's default storage class + replicaIndices: + - 0 + - 2 + - clusterName: demo-worker + storageClassName: local-path # optional; omit to use the cluster's default storage class + replicaIndices: + - 1 + slice: + projectNamespace: kubeslice-demo-distributed-postgres + sliceName: demo-slice + nodeSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway + zoneSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway ``` This policy schedules: -- `postgres-0` and `postgres-2` on `demo-controller`. -- `postgres-1` on `demo-worker`. + +- `postgres-0` and `postgres-2` on `demo-controller` +- `postgres-1` on `demo-worker` Apply the policy on `demo-controller`: ```bash -kubectl apply -f podplacementpolicy.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 6: Create a Distributed Postgres Instance -Define a Postgres custom resource with `spec.distributed: true` and reference the `PlacementPolicy`. Example: +Create the `demo` namespace first: + +```bash +$ kubectl create namespace demo +``` + +Define a Postgres custom resource with `spec.distributed` set to `true` and reference the `PlacementPolicy`. Create a `postgres.yaml` file: ```yaml apiVersion: kubedb.com/v1 @@ -649,79 +753,78 @@ spec: Apply the resource on `demo-controller`: ```bash -kubectl apply -f postgres.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f postgres.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 7: Verify the Deployment -1. **Check Postgres Resource and Pods on `demo-controller`**: +#### 1. Check Postgres Resource and Pods on `demo-controller` ```bash -kubectl get pg,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl get pg,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config ``` -**Output**: - ```shell - NAME VERSION STATUS AGE - postgres.kubedb.com/postgres 11.5.2 Ready 99s - - NAME READY STATUS RESTARTS AGE - pod/postgres-0 3/3 Running 0 95s - pod/postgres-2 3/3 Running 0 95s - - NAME TYPE DATA AGE - secret/postgres-auth kubernetes.io/basic-auth 2 95s - - ``` +**Output:** +```bash +NAME VERSION STATUS AGE +postgres.kubedb.com/postgres 17.2 Ready 99s -2. **Check Pods and Secrets on `demo-worker`**: +NAME READY STATUS RESTARTS AGE +pod/postgres-0 2/2 Running 0 95s +pod/postgres-2 2/2 Running 0 95s -```bash -kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config +NAME TYPE DATA AGE +secret/postgres-auth kubernetes.io/basic-auth 2 95s ``` -**Output**: +#### 2. Check Pods and Secrets on `demo-worker` + +```bash +$ kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config ``` -NAME READY STATUS RESTARTS AGE -postgres-1 3/3 Running 0 95s -NAME TYPE DATA AGE -secret/postgres-auth kubernetes.io/basic-auth 2 95s +**Output:** + +```bash +NAME READY STATUS RESTARTS AGE +pod/postgres-1 2/2 Running 0 95s + +NAME TYPE DATA AGE +secret/postgres-auth kubernetes.io/basic-auth 2 95s ``` -3. **Verify Cluster Status**: - Connect to a Postgres pod and check cluster status using SQL queries or KubeDB status fields. +#### 3. Verify Replication Status -```shell -➤ kubectl exec -it -n demo ha-postgres-1 -- bash +Connect to the primary Postgres pod and check the replication status: + +```bash +$ kubectl exec -it -n demo pod/postgres-0 --context demo-controller -- bash Defaulted container "postgres" out of: postgres, pg-coordinator, postgres-init-container (init) -ha-postgres-1:/$ psql -psql (16.8) -Type "help" for help. +postgres-0:/$ psql -U postgres +``` + +Run the following query: -postgres=# select * from pg_stat_replication; +```sql +SELECT * FROM pg_stat_replication; +``` + +**Output:** + +``` pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state | reply_time ------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------------+----------------+-----------------+---------------+------------+------------------------------- - 495 | 10 | postgres | ha-postgres-0 | 10.42.0.78 | | 49492 | 2025-08-26 12:03:47.940257+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000212 | 00:00:00.00207 | 00:00:00.00212 | 0 | async | 2025-08-26 12:05:33.011941+00 - 1183 | 10 | postgres | ha-postgres-2 | 10.42.0.82 | | 58254 | 2025-08-26 12:05:28.257383+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000179 | 00:00:00.00185 | 00:00:00.001912 | 0 | async | 2025-08-26 12:05:33.01175+00 + 495 | 10 | postgres | postgres-1 | 10.1.0.12 | | 49492 | 2025-08-26 12:03:47.940257+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000212 | 00:00:00.00207 | 00:00:00.00212 | 0 | async | 2025-08-26 12:05:33.011941+00 + 1183 | 10 | postgres | postgres-2 | 10.42.0.82 | | 58254 | 2025-08-26 12:05:28.257383+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000179 | 00:00:00.00185 | 00:00:00.001912 | 0 | async | 2025-08-26 12:05:33.01175+00 (2 rows) - - ``` -## Troubleshooting Tips - -- **Pods Not Running**: Check pod logs for errors related to storage, networking, or configuration. -- **Cluster Not Joining**: Ensure the `RawFeedbackJsonString` feature gate is enabled and verify network connectivity between clusters. -- **KubeSlice Issues**: Confirm that the network interface matches your cluster's configuration and that sidecar containers are injected. -- **Postgres Not Synced**: Check replication status and ensure all nodes are part of the cluster. +Both `postgres-1` (on `demo-worker`, connected via KubeSlice) and `postgres-2` (on `demo-controller`) should appear in streaming replication state. ## Next Steps -- **Accessing the Database**: Use the generated secret to retrieve credentials and connect to the Postgres instance. +- **Accessing the Database**: Use the `postgres-auth` secret to retrieve credentials and connect to the Postgres instance. - **Scaling**: Adjust the `PlacementPolicy` to add or remove replicas across clusters. - **Monitoring**: Integrate KubeDB with monitoring tools like Prometheus for cluster health insights. -For further details, refer to the [KubeDB Documentation](https://kubedb.com/docs/v2025.7.31/) - From e15ea7e9f3a0b830d27c85e808b030619bd043b3 Mon Sep 17 00:00:00 2001 From: Bonusree Date: Fri, 12 Jun 2026 17:53:19 +0600 Subject: [PATCH 6/7] delete verify Signed-off-by: Bonusree --- docs/guides/mariadb/distributed/opsrequest/index.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/guides/mariadb/distributed/opsrequest/index.md b/docs/guides/mariadb/distributed/opsrequest/index.md index ae4f0052b9..8c9e36544f 100644 --- a/docs/guides/mariadb/distributed/opsrequest/index.md +++ b/docs/guides/mariadb/distributed/opsrequest/index.md @@ -320,17 +320,6 @@ NAME READY STATUS RESTARTS AGE mariadb-1 3/3 Running 0 20m ``` -Let's verify the cluster size from inside a MariaDB pod: - -```bash -$ kubectl exec -it -n demo mariadb-0 -c mariadb --context demo-controller -- bash -root@mariadb-0:/ mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "show status like 'wsrep_cluster_size';" -+--------------------+-------+ -| Variable_name | Value | -+--------------------+-------+ -| wsrep_cluster_size | 3 | -+--------------------+-------+ -``` From all the above outputs we can see that the cluster now has `3` replicas. We have successfully scaled down the distributed MariaDB cluster. From 24d8d6553a9ebf1806948c78ac742f3e8516f570 Mon Sep 17 00:00:00 2001 From: Bonusree Date: Mon, 15 Jun 2026 11:20:46 +0600 Subject: [PATCH 7/7] image, weight, security issues changes Signed-off-by: Bonusree --- .../autoscaler/storage/overview/index.md | 2 +- .../mariadb/distributed/opsrequest/_index.md | 6 ++-- .../opsrequest/examples/mariadb.yaml | 31 +++++++++++++++++++ .../opsrequest/examples/mdops-downscale.yaml | 11 +++++++ .../opsrequest/examples/mdops-upscale.yaml | 11 +++++++ .../opsrequest/examples/placement-policy.yaml | 27 ++++++++++++++++ .../{index.md => horizontal_scale.md} | 20 +++--------- .../overview/yamls/controller.yaml | 2 +- .../yamls/sliceoperator-controller.yaml | 12 +++---- .../overview/yamls/sliceoperator-worker.yaml | 12 +++---- 10 files changed, 102 insertions(+), 32 deletions(-) create mode 100644 docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml create mode 100644 docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml create mode 100644 docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml create mode 100644 docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml rename docs/guides/mariadb/distributed/opsrequest/{index.md => horizontal_scale.md} (91%) diff --git a/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md b/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md index 7e0abb4430..27ffedfa70 100644 --- a/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md +++ b/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md @@ -29,7 +29,7 @@ This guide will give an overview on how KubeDB Autoscaler operator autoscales th The following diagram shows how KubeDB Autoscaler operator autoscales the resources of `MariaDB` database components. Open the image in a new tab to see the enlarged version.
- Storage Autoscaling process of MariaDB + Storage Autoscaling process of MariaDB
Fig: Storage Autoscaling process of MariaDB
diff --git a/docs/guides/mariadb/distributed/opsrequest/_index.md b/docs/guides/mariadb/distributed/opsrequest/_index.md index cef45b7cf9..1e2027a605 100644 --- a/docs/guides/mariadb/distributed/opsrequest/_index.md +++ b/docs/guides/mariadb/distributed/opsrequest/_index.md @@ -1,10 +1,10 @@ --- -title: Horizontal Scaling +title: Horizontal Scaling menu: docs_{{ .version }}: identifier: guides-mariadb-distributed-horizontalscaling - name: Autoscaling + name: Horizontal Scaling parent: guides-mariadb-distributed - weight: 47 + weight: 50 menu_name: docs_{{ .version }} --- diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml new file mode 100644 index 0000000000..7b16ab9dae --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml @@ -0,0 +1,31 @@ +apiVersion: kubedb.com/v1 +kind: MariaDB +metadata: + name: mariadb + namespace: demo +spec: + version: "11.5.2" + distributed: true + replicas: 3 + storageType: Durable + storage: + storageClassName: "longhorn" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + podTemplate: + spec: + podPlacementPolicy: + name: distributed-mariadb + containers: + - name: mariadb + resources: + requests: + cpu: "200m" + memory: "300Mi" + limits: + cpu: "200m" + memory: "300Mi" + deletionPolicy: WipeOut diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml new file mode 100644 index 0000000000..987979be39 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member: 3 diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml new file mode 100644 index 0000000000..e3fb713195 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-up + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member: 5 diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml new file mode 100644 index 0000000000..2083db243d --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml @@ -0,0 +1,27 @@ +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + labels: + app.kubernetes.io/managed-by: Helm + name: distributed-mariadb +spec: + clusterSpreadConstraint: + distributionRules: + - clusterName: demo-controller + replicaIndices: + - 0 + - 2 + - 4 + - clusterName: demo-worker + replicaIndices: + - 1 + - 3 + slice: + projectNamespace: kubeslice-demo-distributed-mariadb + sliceName: demo-slice + nodeSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway + zoneSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway diff --git a/docs/guides/mariadb/distributed/opsrequest/index.md b/docs/guides/mariadb/distributed/opsrequest/horizontal_scale.md similarity index 91% rename from docs/guides/mariadb/distributed/opsrequest/index.md rename to docs/guides/mariadb/distributed/opsrequest/horizontal_scale.md index 8c9e36544f..cd69825028 100644 --- a/docs/guides/mariadb/distributed/opsrequest/index.md +++ b/docs/guides/mariadb/distributed/opsrequest/horizontal_scale.md @@ -14,7 +14,7 @@ section_menu_id: guides # Horizontal Scale Distributed MariaDB -This guide will show you how to use `KubeDB` Enterprise operator to horizontally scale a distributed MariaDB Galera cluster across multiple Kubernetes clusters. +This guide will show you how to use `KubeDB` Opsrequest operator to horizontally scale a distributed MariaDB Galera cluster across multiple Kubernetes clusters. > **Note:** All other `OpsRequest` operations behave consistently in a distributed environment. Only `HorizontalScaling` has cluster-specific considerations covered in this guide. @@ -47,7 +47,7 @@ Here, we are going to deploy a distributed `MariaDB` Galera cluster and then app ### Deploy PlacementPolicy -For a distributed MariaDB cluster, the `PlacementPolicy` must be created **before** deploying the database. It defines which replica index is scheduled on which cluster and acts as the upper bound for scaling — the actual running replicas can be any number less than or equal to the total indices defined in the policy. +For a distributed MariaDB cluster, the `PlacementPolicy` must be created before deploying the database. It defines which replica index is scheduled on which cluster and acts as the upper bound for scaling, the actual running replicas can be any number less than or equal to the total indices defined in the policy. In this example, the `PlacementPolicy` is configured with **five** replica indices distributed across two clusters. The MariaDB cluster will start with **three** replicas (indices `0`, `1`, `2`), and can be scaled up to **five** (indices `0`–`4`) without modifying the policy. @@ -94,7 +94,7 @@ placementpolicy.apps.k8s.appscode.com/distributed-mariadb created ### Deploy Distributed MariaDB Cluster -In this section, we are going to deploy a distributed MariaDB Galera cluster with version `11.5.2`. Note that `spec.distributed` is set to `true` and `spec.replicas` is `3` — which is less than the five indices defined in the `PlacementPolicy`. The operator will only create pods for indices `0`, `1`, and `2`. +In this section, we are going to deploy a distributed MariaDB Galera cluster with version `11.5.2`. Note that `spec.distributed` is set to `true` and `spec.replicas` is `3` which is less than the five indices defined in the `PlacementPolicy`. The operator will only create pods for indices `0`, `1`, and `2`. ```yaml apiVersion: kubedb.com/v1 @@ -142,7 +142,7 @@ Now, wait until `mariadb` has status `Ready`: ```bash $ kubectl get mariadb -n demo --context demo-controller NAME VERSION STATUS AGE -mariadb 11.5.2 Ready 2m36s +mariadb 11.5.2 Ready 2m36s ``` Let's check the number of replicas this database has from the MariaDB object: @@ -165,19 +165,9 @@ NAME READY STATUS RESTARTS AGE mariadb-1 3/3 Running 0 2m30s ``` -We can also verify the cluster size from inside a MariaDB pod: -```bash -$ kubectl exec -it -n demo mariadb-0 -c mariadb --context demo-controller -- bash -root@mariadb-0:/ mysql -uroot -p$MYSQL_ROOT_PASSWORD -e "show status like 'wsrep_cluster_size';" -+--------------------+-------+ -| Variable_name | Value | -+--------------------+-------+ -| wsrep_cluster_size | 3 | -+--------------------+-------+ -``` -We can see the cluster has 3 nodes. We are now ready to apply `MariaDBOpsRequest` to scale this database. +We can see the cluster has 3 pods. We are now ready to apply `MariaDBOpsRequest` to scale this database. ## Scale Up Replicas diff --git a/docs/guides/mariadb/distributed/overview/yamls/controller.yaml b/docs/guides/mariadb/distributed/overview/yamls/controller.yaml index 423bc924b4..330b6ac4e1 100644 --- a/docs/guides/mariadb/distributed/overview/yamls/controller.yaml +++ b/docs/guides/mariadb/distributed/overview/yamls/controller.yaml @@ -3,4 +3,4 @@ kubeslice: loglevel: info rbacResourcePrefix: kubeslice-rbac projectnsPrefix: kubeslice - endpoint: https://10.2.0.56:6443 \ No newline at end of file + endpoint: https://:6443 \ No newline at end of file diff --git a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml index de096e48eb..502cb6ee9f 100644 --- a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml +++ b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml @@ -1,11 +1,11 @@ ## Base64 encoded secret values from controller cluster controllerSecret: - namespace: a3ViZXNsaWNlLWRlbW8tZGlzdHJpYnV0ZWQtbWFyaWFkYg== - endpoint: aHR0cHM6Ly8xMC4yLjAuNTY6NjQ0Mw== - ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3SGhjTk1qVXdPREEyTURVeE5ERXhXaGNOTXpVd09EQTBNRFV4TkRFeApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRZ0d0VVc3bFA5aWZLajNzN01rZmFwU1NxZFptYXJaN0tsYjBzZmIxUksKU2tkMkR5YVB2Q01BQkZoZ2EvRlJSd3pIZGxCL3kxMHEvcUtGNm85VXBKMjdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWlRNkxlekFRbERGSHF3SndxVHpFClpnNGxzTTh3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQUlEUjlwZmZYcWFqd0VXd3U2cWpYVkFmNkNvVGZaRXEKa0NUN1dMOXZ1NjErQWlBOHhFTFVxSXNHSXc1eTlQM21rRnVHdDQzNGJDYkhraDF6OHJQT3RsZ2tDUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K - token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrZzJObEYyY0RKVlZGUnlOVVI1VFRJM04wazRORzFhV1ZSM2IwMTVTbnBSU2psTE1UQXpTa2RJUkdNaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpyZFdKbGMyeHBZMlV0WkdWdGJ5MWthWE4wY21saWRYUmxaQzF0WVhKcFlXUmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbXQxWW1WemJHbGpaUzF5WW1GakxYZHZjbXRsY2kxa1pXMXZMV052Ym5SeWIyeHNaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pYTNWaVpYTnNhV05sTFhKaVlXTXRkMjl5YTJWeUxXUmxiVzh0WTI5dWRISnZiR3hsY2lJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExuVnBaQ0k2SWpJeE9UTmtNamd4TFRRMVl6Z3RORGcxTlMwNU56QXdMVEF4T1RJM04yRmhaakpqWlNJc0luTjFZaUk2SW5ONWMzUmxiVHB6WlhKMmFXTmxZV05qYjNWdWREcHJkV0psYzJ4cFkyVXRaR1Z0Ynkxa2FYTjBjbWxpZFhSbFpDMXRZWEpwWVdSaU9tdDFZbVZ6YkdsalpTMXlZbUZqTFhkdmNtdGxjaTFrWlcxdkxXTnZiblJ5YjJ4c1pYSWlmUS5ZU1c3Sjl1N2NtUjVBTWhmMHY1ZVFtdmVpTUh5VlVJelBfTXBqV1NfY0pwcG0yWXJUOWZSNHRqejR4MGx3OHlSR2hrb1JuUGJXLXVMU1pDOExWNm0zX2Zxa0dHN3l3MFhQM3hCOWJsOEdxaGNVaG1rd0JCWHEtbEEybkZ6T0MtVTRqMEhOMnlRS0EtdzJQRUE4dnFGUlByUWVLckJwN0pLRXFUOFExbHMtTUNiUWNYZHJ0UDVNN255QXhSTHVsNnZJRlFkM0cwZU1RMzMwU3JNVlVsV2FaZ0NSbmJHZ2FGbnlwdks2RnlLZG9XUzg2ZzR6Sk1hZ0NRY2N3QnNibEN2anJEUHR4X1h6cVo2RWwwblpYanZTTFEyOGJGdU5DdmJ1QlI1T1JGSzI2aVZyQ3MxNnJYUlpSM2NTQXh3MTN0NDRGQVBraWg3ZlRJUEV6bjhnN0RTV0E= + namespace: + endpoint: + ca.crt: + token: cluster: name: demo-controller - endpoint: https://10.2.0.56:6443 + endpoint: https://:6443 netop: - networkInterface: enp1s0 + networkInterface: diff --git a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml index 9ff1a6eca4..ffd8327a7b 100644 --- a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml +++ b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml @@ -1,11 +1,11 @@ ## Base64 encoded secret values from controller cluster controllerSecret: - namespace: a3ViZXNsaWNlLWRlbW8tZGlzdHJpYnV0ZWQtbWFyaWFkYg== - endpoint: aHR0cHM6Ly8xMC4yLjAuNTY6NjQ0Mw== - ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3SGhjTk1qVXdPREEyTURVeE5ERXhXaGNOTXpVd09EQTBNRFV4TkRFeApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRZ0d0VVc3bFA5aWZLajNzN01rZmFwU1NxZFptYXJaN0tsYjBzZmIxUksKU2tkMkR5YVB2Q01BQkZoZ2EvRlJSd3pIZGxCL3kxMHEvcUtGNm85VXBKMjdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWlRNkxlekFRbERGSHF3SndxVHpFClpnNGxzTTh3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQUlEUjlwZmZYcWFqd0VXd3U2cWpYVkFmNkNvVGZaRXEKa0NUN1dMOXZ1NjErQWlBOHhFTFVxSXNHSXc1eTlQM21rRnVHdDQzNGJDYkhraDF6OHJQT3RsZ2tDUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K - token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrZzJObEYyY0RKVlZGUnlOVVI1VFRJM04wazRORzFhV1ZSM2IwMTVTbnBSU2psTE1UQXpTa2RJUkdNaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpyZFdKbGMyeHBZMlV0WkdWdGJ5MWthWE4wY21saWRYUmxaQzF0WVhKcFlXUmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbXQxWW1WemJHbGpaUzF5WW1GakxYZHZjbXRsY2kxa1pXMXZMWGR2Y210bGNpSXNJbXQxWW1WeWJtVjBaWE11YVc4dmMyVnlkbWxqWldGalkyOTFiblF2YzJWeWRtbGpaUzFoWTJOdmRXNTBMbTVoYldVaU9pSnJkV0psYzJ4cFkyVXRjbUpoWXkxM2IzSnJaWEl0WkdWdGJ5MTNiM0pyWlhJaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNTFhV1FpT2lJMU9URmxNR0UyWlMwd01EWmpMVFJrWkdZdFlqVmhNQzA0TkRsbVpqQTBOalkzTTJFaUxDSnpkV0lpT2lKemVYTjBaVzA2YzJWeWRtbGpaV0ZqWTI5MWJuUTZhM1ZpWlhOc2FXTmxMV1JsYlc4dFpHbHpkSEpwWW5WMFpXUXRiV0Z5YVdGa1lqcHJkV0psYzJ4cFkyVXRjbUpoWXkxM2IzSnJaWEl0WkdWdGJ5MTNiM0pyWlhJaWZRLkhmaE5Ba1J5VmZCQ0pVV1lLTDN3MXhRdVVNakJzMUZFLVotelNRX3pPNTU3NWttUkczUzhyMUlZV1FJdzNOLWZydzhvUlpTRURwWWhsdnhVVTlxYUFsXzJuZkczVDE5OUZZc2hMNEt4U0JWZlhaT0puaGdaS1ZuOW40MTY2VHAwV0NRVTZ1WE1MZE80TTgwV21neGVORWRzVUtYU05iekVHRWRFY3oteWg3dkRteThQUmwyVFZjUFFTamJkQ0l0UWxBMTQ0STVraWMxSGRCNVdiTHR1WDZ0N2hIaHNOYzlNZkYwZXBBZkd4a0YwVDFTWHdNcDBHY3c5cW52OVEzSk91d2liemdXTGF0aUIyWHQtVEdnNWtSZTlCMzduelBlQi1uTmtRY3FGeWEzb2x2Q01QNDRMcW9CbGdqMTJlRGhjOFB6dEFseXRiQ3Z1S1l3Y0lsYk9PUQ== + namespace: + endpoint: + ca.crt: + token: cluster: name: demo-worker - endpoint: https://10.2.0.60:6443 + endpoint: https://:6443 netop: - networkInterface: enp1s0 + networkInterface: