diff --git a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md index fdce0ac570..3b16841bb9 100644 --- a/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md +++ b/docs/guides/mariadb/distributed/autoscaler/storage/cluster/index.md @@ -46,22 +46,33 @@ namespace/demo created ## Storage Autoscaling of Distributed Cluster Database -At first verify that your clusters have a storage class that supports volume expansion. Let's check, +At first verify that your clusters have a storage class that supports volume expansion. First let's check the storagclass of `Controler` cluster, +```bash +$ kubectl get storageclass --context demo-controller +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 4d +longhorn (default) driver.longhorn.io Delete Immediate true 24h +longhorn-static driver.longhorn.io Delete Immediate true 24h +``` +Then check the storageclass of `Worker` cluster, ```bash $ kubectl get storageclass --context demo-worker -NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE -standard (default) rancher.io/local-path Delete WaitForFirstConsumer false 79m -topolvm-provisioner topolvm.cybozu.com Delete WaitForFirstConsumer true 78m +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 4d +longhorn (default) driver.longhorn.io Delete Immediate true 23h +longhorn-static driver.longhorn.io Delete Immediate true 23h ``` -We can see from the output the `topolvm-provisioner` storage class has `ALLOWVOLUMEEXPANSION` field as true. So, this storage class supports volume expansion. We can use it. You can install topolvm from [here](https://github.com/topolvm/topolvm) +We can see from the output the `longhorn (default)` and `longhorn-static` storage class has `ALLOWVOLUMEEXPANSION` field as true. So, this storage class supports volume expansion. We can use it. You can install `longhorn` from [here](https://longhorn.io/docs/1.11.2/deploy/install/) ### Deploy PlacementPolicy -For distributed MariaDB autoscaling, the `PlacementPolicy` must include a `monitoring.prometheus.url` for each spoke cluster. The autoscaler uses these endpoints to monitor storage usage across all clusters where MariaDB pods are running. +For distributed `MariaDB` autoscaling, the PlacementPolicy must define a valid `monitoring.prometheus.url` for each spoke cluster. These endpoints are required by the autoscaler to collect metrics and monitor storage usage across all clusters where MariaDB pods are deployed. + +Additionally, it is essential to explicitly specify the storage class for each cluster using `spec.clusterSpreadConstraint.distributionRules.storageClassName`. If this field is not provided, the system will fall back to the default storage class, which may lead to unintended behavior or potential data management risks in a distributed environment. -Below is the YAML of the `PlacementPolicy` that we are going to create. It distributes 4 replicas across two clusters and provides the Prometheus endpoint for each: +Below is the YAML of the `PlacementPolicy` that we are going to create. It distributes 3 replicas across two clusters and provides the Prometheus endpoint for each: ```yaml apiVersion: apps.k8s.appscode.com/v1 @@ -74,6 +85,7 @@ spec: clusterSpreadConstraint: distributionRules: - clusterName: demo-controller + storageClassName: local-path monitoring: prometheus: url: http://prometheus-operated.monitoring.svc.cluster.local:9090 @@ -81,6 +93,7 @@ spec: - 0 - 2 - clusterName: demo-worker + storageClassName: local-path monitoring: prometheus: url: http://prometheus-operated.monitoring.svc.cluster.local:9090 diff --git a/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md b/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md index 7e0abb4430..27ffedfa70 100644 --- a/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md +++ b/docs/guides/mariadb/distributed/autoscaler/storage/overview/index.md @@ -29,7 +29,7 @@ This guide will give an overview on how KubeDB Autoscaler operator autoscales th The following diagram shows how KubeDB Autoscaler operator autoscales the resources of `MariaDB` database components. Open the image in a new tab to see the enlarged version.
- Storage Autoscaling process of MariaDB + Storage Autoscaling process of MariaDB
Fig: Storage Autoscaling process of MariaDB
diff --git a/docs/guides/mariadb/distributed/opsrequest/_index.md b/docs/guides/mariadb/distributed/opsrequest/_index.md new file mode 100644 index 0000000000..1e2027a605 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/_index.md @@ -0,0 +1,10 @@ +--- +title: Horizontal Scaling +menu: + docs_{{ .version }}: + identifier: guides-mariadb-distributed-horizontalscaling + name: Horizontal Scaling + parent: guides-mariadb-distributed + weight: 50 +menu_name: docs_{{ .version }} +--- diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml new file mode 100644 index 0000000000..7b16ab9dae --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml @@ -0,0 +1,31 @@ +apiVersion: kubedb.com/v1 +kind: MariaDB +metadata: + name: mariadb + namespace: demo +spec: + version: "11.5.2" + distributed: true + replicas: 3 + storageType: Durable + storage: + storageClassName: "longhorn" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + podTemplate: + spec: + podPlacementPolicy: + name: distributed-mariadb + containers: + - name: mariadb + resources: + requests: + cpu: "200m" + memory: "300Mi" + limits: + cpu: "200m" + memory: "300Mi" + deletionPolicy: WipeOut diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml new file mode 100644 index 0000000000..987979be39 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member: 3 diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml new file mode 100644 index 0000000000..e3fb713195 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml @@ -0,0 +1,11 @@ +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-up + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member: 5 diff --git a/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml b/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml new file mode 100644 index 0000000000..2083db243d --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml @@ -0,0 +1,27 @@ +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + labels: + app.kubernetes.io/managed-by: Helm + name: distributed-mariadb +spec: + clusterSpreadConstraint: + distributionRules: + - clusterName: demo-controller + replicaIndices: + - 0 + - 2 + - 4 + - clusterName: demo-worker + replicaIndices: + - 1 + - 3 + slice: + projectNamespace: kubeslice-demo-distributed-mariadb + sliceName: demo-slice + nodeSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway + zoneSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway diff --git a/docs/guides/mariadb/distributed/opsrequest/horizontal_scale.md b/docs/guides/mariadb/distributed/opsrequest/horizontal_scale.md new file mode 100644 index 0000000000..cd69825028 --- /dev/null +++ b/docs/guides/mariadb/distributed/opsrequest/horizontal_scale.md @@ -0,0 +1,324 @@ +--- +title: Distributed MariaDBOpsRequest +menu: + docs_{{ .version }}: + identifier: guides-mariadb-distributed-opsrequest + name: Distributed MariaDBOpsRequest + parent: guides-mariadb-distributed-horizontalscaling + weight: 20 +menu_name: docs_{{ .version }} +section_menu_id: guides +--- + +> New to KubeDB? Please start [here](/docs/README.md). + +# Horizontal Scale Distributed MariaDB + +This guide will show you how to use `KubeDB` Opsrequest operator to horizontally scale a distributed MariaDB Galera cluster across multiple Kubernetes clusters. + +> **Note:** All other `OpsRequest` operations behave consistently in a distributed environment. Only `HorizontalScaling` has cluster-specific considerations covered in this guide. + +## Before You Begin + +- At first, you need to have a multi-cluster Kubernetes setup with OCM and KubeSlice configured. Follow the [Distributed MariaDB Overview](/docs/guides/mariadb/distributed/overview) guide to set up the required infrastructure. + +- Install `KubeDB` Community and Enterprise operator in your hub cluster following the steps [here](/docs/setup/README.md). Make sure to enable OCM support: + + ```bash + --set petset.features.ocm.enabled=true + ``` + +- You should be familiar with the following `KubeDB` concepts: + - [MariaDB](/docs/guides/mariadb/concepts/mariadb/) + - [Distributed MariaDB Overview](/docs/guides/mariadb/distributed/overview/index.md) + - [MariaDBOpsRequest](/docs/guides/mariadb/concepts/opsrequest/) + - [Horizontal Scaling Overview](/docs/guides/mariadb/scaling/horizontal-scaling/overview/) + +To keep everything isolated, we are going to use a separate namespace called `demo` throughout this tutorial. + +```bash +$ kubectl create ns demo +namespace/demo created +``` + +## Apply Horizontal Scaling on Distributed Cluster + +Here, we are going to deploy a distributed `MariaDB` Galera cluster and then apply horizontal scaling using `MariaDBOpsRequest`. + +### Deploy PlacementPolicy + +For a distributed MariaDB cluster, the `PlacementPolicy` must be created before deploying the database. It defines which replica index is scheduled on which cluster and acts as the upper bound for scaling, the actual running replicas can be any number less than or equal to the total indices defined in the policy. + +In this example, the `PlacementPolicy` is configured with **five** replica indices distributed across two clusters. The MariaDB cluster will start with **three** replicas (indices `0`, `1`, `2`), and can be scaled up to **five** (indices `0`–`4`) without modifying the policy. + +```yaml +apiVersion: apps.k8s.appscode.com/v1 +kind: PlacementPolicy +metadata: + labels: + app.kubernetes.io/managed-by: Helm + name: distributed-mariadb +spec: + clusterSpreadConstraint: + distributionRules: + - clusterName: demo-controller + replicaIndices: + - 0 + - 2 + - 4 + - clusterName: demo-worker + replicaIndices: + - 1 + - 3 + slice: + projectNamespace: kubeslice-demo-distributed-mariadb + sliceName: demo-slice + nodeSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway + zoneSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway +``` + +Here, + +- `spec.clusterSpreadConstraint.distributionRules[].replicaIndices` specifies which MariaDB replica indices are scheduled on that cluster. `demo-controller` will host replicas `0`, `2`, and `4`; `demo-worker` will host replicas `1` and `3`. + +Apply the `PlacementPolicy` on the hub (`demo-controller`) cluster: + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/placement-policy.yaml --context demo-controller +placementpolicy.apps.k8s.appscode.com/distributed-mariadb created +``` + +### Deploy Distributed MariaDB Cluster + +In this section, we are going to deploy a distributed MariaDB Galera cluster with version `11.5.2`. Note that `spec.distributed` is set to `true` and `spec.replicas` is `3` which is less than the five indices defined in the `PlacementPolicy`. The operator will only create pods for indices `0`, `1`, and `2`. + +```yaml +apiVersion: kubedb.com/v1 +kind: MariaDB +metadata: + name: mariadb + namespace: demo +spec: + version: "11.5.2" + distributed: true + replicas: 3 + storageType: Durable + storage: + storageClassName: "longhorn" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + podTemplate: + spec: + podPlacementPolicy: + name: distributed-mariadb + containers: + - name: mariadb + resources: + requests: + cpu: "200m" + memory: "300Mi" + limits: + cpu: "200m" + memory: "300Mi" + deletionPolicy: WipeOut +``` + +Let's create the `MariaDB` CR we have shown above, + +```bash +$ kubectl create -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/mariadb.yaml --context demo-controller +mariadb.kubedb.com/mariadb created +``` + +Now, wait until `mariadb` has status `Ready`: + +```bash +$ kubectl get mariadb -n demo --context demo-controller +NAME VERSION STATUS AGE +mariadb 11.5.2 Ready 2m36s +``` + +Let's check the number of replicas this database has from the MariaDB object: + +```bash +$ kubectl get mariadb -n demo mariadb -o json | jq '.spec.replicas' +3 +``` + +The pods are distributed across clusters as defined by the `PlacementPolicy`. Indices `0` and `2` land on `demo-controller`; index `1` lands on `demo-worker`: + +```bash +$ kubectl get pods -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 2m30s +mariadb-2 3/3 Running 0 2m30s + +$ kubectl get pods -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-1 3/3 Running 0 2m30s +``` + + + +We can see the cluster has 3 pods. We are now ready to apply `MariaDBOpsRequest` to scale this database. + +## Scale Up Replicas + +Here, we are going to scale up the replicas from `3` to `5`. Because the `PlacementPolicy` already defines indices `0`–`4`, no changes to the policy are required — the operator knows where to place the new pods. + +### Create MariaDBOpsRequest + +In order to scale up the replicas, we have to create a `MariaDBOpsRequest` CR with our desired replicas. Below is the YAML of the `MariaDBOpsRequest` CR that we are going to create: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-up + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member : 5 +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing horizontal scaling on `mariadb`. +- `spec.type` specifies that we are performing `HorizontalScaling`. +- `spec.horizontalScaling.member` specifies the desired replica count after scaling. This value must not exceed the total number of indices defined in the `PlacementPolicy` (5 in this example). + +Let's create the `MariaDBOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/mdops-upscale.yaml --context demo-controller +mariadbopsrequest.ops.kubedb.com/mdops-scale-horizontal-up created +``` + +### Verify Cluster Replicas Scaled Up Successfully + +If everything goes well, `KubeDB` Enterprise operator will update the replicas of the `MariaDB` object and create the new pods on the clusters specified by the `PlacementPolicy`. + +Let's wait for `MariaDBOpsRequest` to be `Successful`: + +```bash +$ watch kubectl get mariadbopsrequest -n demo --context demo-controller +Every 2.0s: kubectl get mariadbopsrequest -n demo +NAME TYPE STATUS AGE +mdops-scale-horizontal-up HorizontalScaling Successful 18m +``` + +We can see from the above output that the `MariaDBOpsRequest` has succeeded. Now, let's verify the number of replicas: + +```bash +$ kubectl get mariadb -n demo mariadb -o json | jq '.spec.replicas' +5 +``` + +The two new pods are placed on the clusters according to the `PlacementPolicy` — index `4` on `demo-controller` and index `3` on `demo-worker`: + +```bash +$ kubectl get pods -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 58m +mariadb-2 3/3 Running 0 55m +mariadb-4 3/3 Running 0 17m + + +$ kubectl get pods -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-1 3/3 Running 0 57m +mariadb-3 3/3 Running 0 19m +``` + +From all the above outputs we can see that the cluster now has `5` replicas. We have successfully scaled up the distributed MariaDB cluster. + +## Scale Down Replicas + +Here, we are going to scale down the replicas from `5` to `3`. The `PlacementPolicy` does not need to be modified — the operator will remove the highest-indexed pods (`mariadb-3` and `mariadb-4`) and the remaining pods will continue running as defined by the policy. + +### Create MariaDBOpsRequest + +In order to scale down the cluster, we have to create a `MariaDBOpsRequest` CR with our desired replicas. Below is the YAML of the `MariaDBOpsRequest` CR that we are going to create: + +```yaml +apiVersion: ops.kubedb.com/v1alpha1 +kind: MariaDBOpsRequest +metadata: + name: mdops-scale-horizontal-down + namespace: demo +spec: + type: HorizontalScaling + databaseRef: + name: mariadb + horizontalScaling: + member: 3 +``` + +Here, + +- `spec.databaseRef.name` specifies that we are performing horizontal scaling down on `mariadb`. +- `spec.type` specifies that we are performing `HorizontalScaling`. +- `spec.horizontalScaling.member` specifies the desired replica count after scaling. + +Let's create the `MariaDBOpsRequest` CR we have shown above, + +```bash +$ kubectl apply -f https://github.com/kubedb/docs/raw/{{< param "info.version" >}}/docs/guides/mariadb/distributed/opsrequest/examples/mdops-downscale.yaml --context demo-controller +mariadbopsrequest.ops.kubedb.com/mdops-scale-horizontal-down created +``` + +### Verify Cluster Replicas Scaled Down Successfully + +If everything goes well, `KubeDB` Enterprise operator will update the replicas of the `MariaDB` object and remove the highest-indexed pods from their respective clusters. + +Let's wait for `MariaDBOpsRequest` to be `Successful`: + +```bash +$ watch kubectl get mariadbopsrequest -n demo --context demo-controller +Every 2.0s: kubectl get mariadbopsrequest -n demo +NAME TYPE STATUS AGE +mdops-scale-horizontal-down HorizontalScaling Successful 2m32s +``` + +We can see from the above output that the `MariaDBOpsRequest` has succeeded. Now, let's verify the number of replicas: + +```bash +$ kubectl get mariadb -n demo mariadb -o json | jq '.spec.replicas' +3 +``` + +Pods `mariadb-3` and `mariadb-4` have been removed from their respective clusters: + +```bash +$ kubectl get pods -n demo --context demo-controller +NAME READY STATUS RESTARTS AGE +mariadb-0 3/3 Running 0 20m +mariadb-2 3/3 Running 0 20m + +$ kubectl get pods -n demo --context demo-worker +NAME READY STATUS RESTARTS AGE +mariadb-1 3/3 Running 0 20m +``` + + +From all the above outputs we can see that the cluster now has `3` replicas. We have successfully scaled down the distributed MariaDB cluster. + +## Cleaning Up + +To clean up the Kubernetes resources created by this tutorial, run: + +```bash +$ kubectl delete mariadb -n demo mariadb --context demo-controller +$ kubectl delete mariadbopsrequest -n demo mdops-scale-horizontal-up mdops-scale-horizontal-down --context demo-controller +$ kubectl delete placementpolicy distributed-mariadb --context demo-controller +``` diff --git a/docs/guides/mariadb/distributed/overview/index.md b/docs/guides/mariadb/distributed/overview/index.md index f2ffe93a07..34e8fe0f01 100644 --- a/docs/guides/mariadb/distributed/overview/index.md +++ b/docs/guides/mariadb/distributed/overview/index.md @@ -51,12 +51,12 @@ Follow these steps to deploy a distributed MariaDB Galera cluster across multipl Ensure your `KUBECONFIG` is set up to switch between clusters. This guide uses two clusters: `demo-controller` (hub and spoke) and `demo-worker` (spoke). ```bash -kubectl config get-contexts +$ kubectl config get-contexts ``` **Output:** -``` +```bash CURRENT NAME CLUSTER AUTHINFO NAMESPACE * demo-controller demo-controller demo-controller demo-worker demo-worker demo-worker @@ -67,8 +67,8 @@ CURRENT NAME CLUSTER AUTHINFO NAMESPACE On the `demo-controller` cluster, initialize the OCM hub: ```bash -kubectl config use-context demo-controller -clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true +$ kubectl config use-context demo-controller +$ clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true ``` #### 3. Verify Hub Deployment @@ -76,7 +76,7 @@ clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true Check the pods in the `open-cluster-management-hub` namespace to ensure all components are running: ```bash -kubectl get pods -n open-cluster-management-hub +$ kubectl get pods -n open-cluster-management-hub ``` **Output:** @@ -98,7 +98,7 @@ All pods should be in the `Running` state with `1/1` readiness and no restarts, Obtain the join token from the hub cluster: ```bash -clusteradm get token +$ clusteradm get token ``` **Output:** @@ -106,14 +106,14 @@ clusteradm get token ``` token= please log on spoke and run: -clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name +clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name ``` On the `demo-worker` cluster, join it to the hub. Include the `RawFeedbackJsonString` feature gate for resource feedback: ```bash -kubectl config use-context demo-worker -clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true +$ kubectl config use-context demo-worker +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true ``` #### 5. Accept Spoke Cluster @@ -121,8 +121,8 @@ clusteradm join --hub-token --hub-apiserver https:/ On the `demo-controller` cluster, accept the `demo-worker` cluster: ```bash -kubectl config use-context demo-controller -clusteradm accept --clusters demo-worker +$ kubectl config use-context demo-controller +$ clusteradm accept --clusters demo-worker ``` > **Note:** It may take a few attempts (e.g., retry every 10 seconds) if the cluster is not immediately available. @@ -141,12 +141,12 @@ Your managed cluster demo-worker has joined the Hub successfully. Confirm that a namespace for `demo-worker` was created on the hub cluster: ```bash -kubectl get ns +$ kubectl get ns ``` **Output:** -``` +```bash NAME STATUS AGE default Active 99m demo-worker Active 58s @@ -162,15 +162,15 @@ open-cluster-management-hub Active 5m32s Repeat the join and accept process for `demo-controller` so it can also act as a spoke cluster: ```bash -kubectl config use-context demo-controller -clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true -clusteradm accept --clusters demo-controller +$ kubectl config use-context demo-controller +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true +$ clusteradm accept --clusters demo-controller ``` Verify the namespace for `demo-controller`: ```bash -kubectl get ns +$ kubectl get ns ``` **Output:** @@ -189,31 +189,50 @@ open-cluster-management-agent-addon Active 34s open-cluster-management-hub Active 10m ``` -### Step 2: Configure OCM WorkConfiguration (Optional) +#### 8. Verify OCM Roles -If you did not follow the provided OCM installation steps, update the `klusterlet` resource to enable feedback retrieval: +After registration, use these commands to confirm which cluster is the hub and which are spokes: ```bash -kubectl edit klusterlet klusterlet +# Hub: lists all registered spoke clusters +$ kubectl get managedclusters + +# Spoke: shows this cluster's registered name +$ kubectl get klusterlet klusterlet -o jsonpath='{.spec.clusterName}' + +# Hub components run only on the hub cluster +$ kubectl get pods -n open-cluster-management-hub + +# Spoke agent runs on every spoke cluster +$ kubectl get pods -n open-cluster-management-agent ``` -Add the following under the `spec` field: +### Step 2: Configure OCM WorkConfiguration -```yaml -workConfiguration: - featureGates: - - feature: RawFeedbackJsonString - mode: Enable - hubKubeAPIBurst: 100 - hubKubeAPIQPS: 50 - kubeAPIBurst: 100 - kubeAPIQPS: 50 +Run this on **every spoke cluster** (`demo-controller` and `demo-worker`). This enables the `RawFeedbackJsonString` feature gate that KubeDB requires to read pod status across clusters, and raises the API rate limits to prevent throttling. + +> **Note:** Even if you passed `--feature-gates=RawFeedbackJsonString=true` during `clusteradm join`, the rate limit fields are not set by that flag. Run this patch on all spokes regardless. +> +> **Why this matters:** KubeDB uses OCM's ManifestWork feedback mechanism to watch the status of MariaDB pods on remote spoke clusters. Without `RawFeedbackJsonString`, the KubeDB provisioner on the hub never receives pod status updates from spokes and the distributed MariaDB CR will stay in a non-Ready state indefinitely. The rate limits prevent the klusterlet agent from being API-throttled during initial cluster formation. + +```bash +$ kubectl patch klusterlet klusterlet --type=merge -p '{ + "spec": { + "workConfiguration": { + "featureGates": [{"feature": "RawFeedbackJsonString", "mode": "Enable"}], + "hubKubeAPIBurst": 100, + "hubKubeAPIQPS": 50, + "kubeAPIBurst": 100, + "kubeAPIQPS": 50 + } + } +}' ``` Verify the configuration: ```bash -kubectl get klusterlet klusterlet -oyaml +$ kubectl get klusterlet klusterlet -oyaml ``` **Sample Output (abridged):** @@ -241,7 +260,13 @@ KubeSlice enables pod-to-pod communication across clusters. Install the KubeSlic #### 1. Install KubeSlice Controller -On `demo-controller`, create a `controller.yaml` file: +On `demo-controller`, get the hub API server address first: + +```bash +$ kubectl cluster-info | grep 'Kubernetes control plane' +``` + +Use the IP and port from that output as the `endpoint` value. Create a `controller.yaml` file: ```yaml kubeslice: @@ -249,13 +274,13 @@ kubeslice: loglevel: info rbacResourcePrefix: kubeslice-rbac projectnsPrefix: kubeslice - endpoint: https://10.2.0.56:6443 + endpoint: https://:6443 ``` Deploy the controller using Helm: ```bash -helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ +$ helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ --version v2026.1.15 \ -f controller.yaml \ --namespace kubeslice-controller \ @@ -267,7 +292,7 @@ helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-con Verify the installation: ```bash -kubectl get pods -n kubeslice-controller +$ kubectl get pods -n kubeslice-controller ``` **Output:** @@ -296,18 +321,18 @@ spec: Apply the project: ```bash -kubectl apply -f project.yaml +$ kubectl apply -f project.yaml ``` Verify: ```bash -kubectl get project -n kubeslice-controller +$ kubectl get project -n kubeslice-controller ``` **Output:** -``` +```bash NAME AGE demo-distributed-mariadb 31s ``` @@ -315,7 +340,7 @@ demo-distributed-mariadb 31s Check service accounts: ```bash -kubectl get sa -n kubeslice-demo-distributed-mariadb +$ kubectl get sa -n kubeslice-demo-distributed-mariadb ``` **Output:** @@ -333,24 +358,24 @@ Assign the `kubeslice.io/node-type=gateway` label to the node where the worker o On `demo-controller`: ```bash -kubectl get nodes -kubectl label node demo-master kubeslice.io/node-type=gateway +$ kubectl get nodes +$ kubectl label node kubeslice.io/node-type=gateway ``` On `demo-worker`: ```bash -kubectl config use-context demo-worker -kubectl get nodes -kubectl label node demo-worker kubeslice.io/node-type=gateway +$ kubectl config use-context demo-worker +$ kubectl get nodes +$ kubectl label node kubeslice.io/node-type=gateway ``` #### 4. Register Clusters with KubeSlice -Identify the network interface for both clusters by running the following command on the node: +Identify the network interface for each cluster by running the following command **on the gateway node of each cluster**: ```bash -ip route get 8.8.8.8 | awk '{ print $5 }' +$ ip route get 8.8.8.8 | awk '{ print $5 }' ``` **Output (example):** @@ -359,11 +384,15 @@ ip route get 8.8.8.8 | awk '{ print $5 }' enp1s0 ``` +> **Important:** The `networkInterface` value must match the primary network interface of each cluster's gateway node. Run the command above on **each cluster separately** and use that cluster's output as its `networkInterface` value. Each cluster may have a **different interface name** — do not assume they are the same. Using the wrong interface name will cause the WireGuard gateway to silently fail and cross-cluster MariaDB replication will never connect. +> +> Example: if `demo-controller` returns `enp3s0` and `demo-worker` returns `eth0`, use those exact values in the YAML below. + Create a `registration.yaml` file: -> **Important:** +> **Note:** > - The cluster name must exactly match the name of the OCM (spoke) cluster. -> - The corresponding `ManagedClusterAddOn` resource must be created in the namespace that bears the same name as the cluster to setup kubeslice worker automatically. +> - The corresponding `ManagedClusterAddOn` resource must be created in the namespace that bears the same name as the cluster to set up the KubeSlice worker automatically. ```yaml apiVersion: controller.kubeslice.io/v1alpha1 @@ -372,7 +401,7 @@ metadata: name: demo-controller namespace: kubeslice-demo-distributed-mariadb spec: - networkInterface: enp1s0 + networkInterface: # replace with output of ip route command clusterProperty: {} --- apiVersion: controller.kubeslice.io/v1alpha1 @@ -381,7 +410,7 @@ metadata: name: demo-worker namespace: kubeslice-demo-distributed-mariadb spec: - networkInterface: enp1s0 + networkInterface: # replace with output of ip route command clusterProperty: {} --- apiVersion: addon.open-cluster-management.io/v1alpha1 @@ -414,30 +443,31 @@ spec: Apply on `demo-controller`: ```bash -kubectl apply -f registration.yaml +$ kubectl apply -f registration.yaml ``` -Verify: +Verify OCM is deploying the KubeSlice worker manifests to each cluster: ```bash -kubectl get clusters -n kubeslice-demo-distributed-mariadb +$ kubectl get managedclusteraddon -A ``` **Output:** ``` -NAME AGE -demo-controller 9s -demo-worker 9s +NAMESPACE NAME AVAILABLE DEGRADED PROGRESSING +demo-controller kubeslice Unknown True +demo-worker kubeslice Unknown True ``` -Verify the worker installation: +`PROGRESSING: True` means OCM is actively deploying. Wait until `kubeslice-operator` shows `2/2 Running` on both clusters before proceeding: ```bash -kubectl get pods -n kubeslice-system +# Run on each spoke cluster +$ kubectl get pods -n kubeslice-system --watch ``` -**Output:** +**Expected output (after KubeSlice worker is fully deployed):** ``` NAME READY STATUS RESTARTS AGE @@ -446,8 +476,6 @@ kubeslice-dns-6bd9749f4d-pvh7g 1/1 Running 0 4m43s kubeslice-install-crds-szhvc 0/1 Completed 0 4m56s kubeslice-netop-g4dfn 1/1 Running 0 4m43s kubeslice-operator-949b7d6f7-9wj7h 2/2 Running 0 4m43s -kubeslice-postdelete-job-ctlzt 0/1 Completed 0 20m -nsm-delete-webhooks-ndksl 0/1 Completed 0 20m nsc-grpc-server-sbjj7 1/1 Running 0 4m43s nsm-install-crds-5z4j9 0/1 Completed 0 4m53s nsmgr-zzwgh 2/2 Running 0 4m43s @@ -503,29 +531,31 @@ spec: Apply the `SliceConfig`: ```bash -kubectl apply -f sliceconfig.yaml +$ kubectl apply -f sliceconfig.yaml ``` +After the SliceConfig is applied, a `vl3-slice-router` pod will appear in `kubeslice-system` on each cluster, indicating the slice VPN tunnel is being established. + #### 6. Configure DNS for KubeSlice -Update the network traffic rules to forward the `*.slice.local` suffix (KubeSlice default domain) to the KubeSlice gateway. +Update CoreDNS to forward `*.slice.local` traffic to the KubeSlice DNS service. Run the following steps on **every cluster** in the slice. -First, get the KubeSlice DNS service IP address: +> **Important:** CoreDNS must be updated and restarted on all clusters before proceeding to Step 4 (KubeDB install). Galera nodes use `.slice.local` DNS names to discover each other across clusters. If DNS is not configured before MariaDB pods start, replication will not form. + +Get the KubeSlice DNS service IP address on each cluster: ```bash -kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' +$ kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' ``` **Output:** -``` +```bash NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR kubeslice-dns ClusterIP 10.43.172.191 53/UDP,53/TCP 8d app=kubeslice-dns ``` -Traffic for `.slice.local` should be forwarded to `10.43.172.191` for this cluster. - -Add the following section to your CoreDNS configuration: +Add the following block to the **top** of your CoreDNS `Corefile` ConfigMap (replace the IP with the one from your cluster): ``` slice.local:53 { @@ -535,10 +565,10 @@ slice.local:53 { } ``` -Example CoreDNS ConfigMap: +Example of the full CoreDNS ConfigMap after editing: ```bash -kubectl get cm -n kube-system coredns -oyaml +$ kubectl get cm -n kube-system coredns -oyaml ``` **Output:** @@ -582,38 +612,54 @@ metadata: namespace: kube-system ``` -> **Note:** Follow the same procedure to update the DNS configuration for all clusters in the slice. +After editing the ConfigMap, restart CoreDNS to apply the change: + +```bash +$ kubectl rollout restart deploy/coredns -n kube-system +``` + +Repeat the DNS configuration steps on every cluster in the slice. ### Step 4: Install the KubeDB Operator -Install the KubeDB Operator on the `demo-controller` cluster to manage the MariaDB instance. +> **Note:** Install the KubeDB operator only on the hub cluster (`demo-controller`). The operator manages MariaDB pods on spoke clusters through OCM — no KubeDB installation is needed on `demo-worker`. #### Get a Free License -Download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb). Obtain the license for the `demo-controller` cluster. +The KubeDB license is tied to the `kube-system` namespace UID of the hub cluster and has an expiry date. Get your cluster UID and verify the license before installing: ```bash -helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ +# Get your cluster UID (required when requesting the license) +$ kubectl get ns kube-system -o jsonpath='{.metadata.uid}' + +# Verify the license is not expired +$ openssl x509 -noout -enddate -in $HOME/Downloads/kubedb-license-.txt +``` + +If expired or not yet obtained, download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb) using the cluster UID above. + +```bash +$ helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ --version v2026.2.26 \ --namespace kubedb --create-namespace \ - --set-file global.license=$HOME/Downloads/kubedb-license-cd548cce-5141-4ed3-9276-6d9578707f12.txt \ + --set-file global.license=$HOME/Downloads/kubedb-license-.txt \ --set petset.features.ocm.enabled=true \ --wait --burst-limit=10000 --debug ``` > **Note:** The `--set petset.features.ocm.enabled=true` flag must be set to enable the MariaDB Distributed feature. -For additional details, refer to the [KubeDB Installation Guide](https://kubedb.com/docs/v2025.6.30/setup/install/kubedb/). +For additional details, refer to the [KubeDB Installation Guide](https://kubedb.com/docs/v2026.4.27/setup/). Verify that the pods are running: ```bash -kubectl get pods -n kubedb +$ kubectl get pods -n kubedb ``` **Output:** -``` +```bash NAME READY STATUS RESTARTS AGE kubedb-kubedb-autoscaler-0 1/2 Running 0 44s kubedb-kubedb-ops-manager-0 1/2 Running 0 43s @@ -625,7 +671,9 @@ kubedb-sidekick-5dbf7bcf64-4b8cw 2/2 Running 0 44s ### Step 5: Define a PlacementPolicy -Create a `PlacementPolicy` to control pod distribution across clusters. Create a `pod-placement-policy.yaml` file: +You can define the `storageClassName` under `spec.clusterSpreadConstraint.distributionRules` for each cluster. If not explicitly specified, the clusters will automatically select an appropriate storage class. Additionally, you can control replica placement by defining which replicas belong to which cluster. When scaling the number of replicas, the distribution must also be specified at the cluster level. In this example, we are using three replicas. + +To manage pod distribution across clusters, create a `PlacementPolicy`. For this purpose, define a `pod-placement-policy.yaml` file as shown below: ```yaml apiVersion: apps.k8s.appscode.com/v1 @@ -638,10 +686,12 @@ spec: clusterSpreadConstraint: distributionRules: - clusterName: demo-controller + storageClassName: local-path # optional; omit to use the cluster's default storage class replicaIndices: - 0 - 2 - clusterName: demo-worker + storageClassName: local-path # optional; omit to use the cluster's default storage class replicaIndices: - 1 slice: @@ -663,11 +713,17 @@ This policy schedules: Apply the policy on `demo-controller`: ```bash -kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 6: Create a Distributed MariaDB Instance +Create the `demo` namespace first: + +```bash +$ kubectl create namespace demo +``` + Define a MariaDB custom resource with `spec.distributed` set to `true` and reference the `PlacementPolicy`. Create a `mariadb.yaml` file: ```yaml @@ -697,7 +753,7 @@ spec: Apply the resource on `demo-controller`: ```bash -kubectl apply -f mariadb.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f mariadb.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 7: Verify the Deployment @@ -705,12 +761,12 @@ kubectl apply -f mariadb.yaml --context demo-controller --kubeconfig $HOME/.kube #### 1. Check MariaDB Resource and Pods on `demo-controller` ```bash -kubectl get md,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl get md,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config ``` **Output:** -``` +```bash NAME VERSION STATUS AGE mariadb.kubedb.com/mariadb 11.5.2 Ready 99s @@ -725,12 +781,12 @@ secret/mariadb-auth kubernetes.io/basic-auth 2 95s #### 2. Check Pods and Secrets on `demo-worker` ```bash -kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config +$ kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config ``` **Output:** -``` +```bash NAME READY STATUS RESTARTS AGE mariadb-1 3/3 Running 0 95s @@ -743,7 +799,7 @@ secret/mariadb-auth kubernetes.io/basic-auth 2 95s Connect to a MariaDB pod and check the Galera cluster status. The primary service DNS follows the format `..svc`: ```bash -kubectl exec -it -n demo pod/mariadb-0 --context demo-controller -- bash +$ kubectl exec -it -n demo pod/mariadb-0 --context demo-controller -- bash mariadb -uroot -p$MYSQL_ROOT_PASSWORD -hmariadb.demo.svc ``` @@ -755,7 +811,7 @@ SHOW STATUS LIKE 'wsrep_cluster_status'; **Output:** -``` +```bash +----------------------+---------+ | Variable_name | Value | +----------------------+---------+ @@ -770,20 +826,7 @@ Check additional Galera status variables: SHOW STATUS LIKE 'wsrep%'; ``` -**Key Indicators:** -- `wsrep_cluster_status: Primary` — The cluster is fully operational. -- `wsrep_cluster_size: 3` — All three nodes are part of the cluster. -- `wsrep_connected: ON` — The node is connected to the cluster. -- `wsrep_ready: ON` — The node is ready to accept queries. -- `wsrep_incoming_addresses` — Lists the IP addresses of all nodes (e.g., `10.1.0.3:0,10.1.0.4:0,10.1.16.4:0`). - -## Troubleshooting - -- **Pods Not Running**: Check pod logs (`kubectl logs -n demo mariadb-0`) for errors related to storage, networking, or configuration. -- **Cluster Not Joining**: Ensure the `RawFeedbackJsonString` feature gate is enabled and verify network connectivity between clusters. -- **KubeSlice Issues**: Confirm that the network interface (e.g., `enp1s0`) matches your cluster's configuration and that sidecar containers are injected. -- **MariaDB Not Synced**: Check `wsrep_local_state_comment` (should be `Synced`) and ensure all nodes have the same `wsrep_cluster_state_uuid`. ## Next Steps @@ -792,4 +835,3 @@ SHOW STATUS LIKE 'wsrep%'; - **Monitoring**: Integrate KubeDB with monitoring tools like Prometheus for cluster health insights. For further details, refer to the [KubeDB Documentation](https://kubedb.com/docs/v2025.7.31/). - diff --git a/docs/guides/mariadb/distributed/overview/yamls/controller.yaml b/docs/guides/mariadb/distributed/overview/yamls/controller.yaml index 423bc924b4..330b6ac4e1 100644 --- a/docs/guides/mariadb/distributed/overview/yamls/controller.yaml +++ b/docs/guides/mariadb/distributed/overview/yamls/controller.yaml @@ -3,4 +3,4 @@ kubeslice: loglevel: info rbacResourcePrefix: kubeslice-rbac projectnsPrefix: kubeslice - endpoint: https://10.2.0.56:6443 \ No newline at end of file + endpoint: https://:6443 \ No newline at end of file diff --git a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml index de096e48eb..502cb6ee9f 100644 --- a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml +++ b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-controller.yaml @@ -1,11 +1,11 @@ ## Base64 encoded secret values from controller cluster controllerSecret: - namespace: a3ViZXNsaWNlLWRlbW8tZGlzdHJpYnV0ZWQtbWFyaWFkYg== - endpoint: aHR0cHM6Ly8xMC4yLjAuNTY6NjQ0Mw== - ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3SGhjTk1qVXdPREEyTURVeE5ERXhXaGNOTXpVd09EQTBNRFV4TkRFeApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRZ0d0VVc3bFA5aWZLajNzN01rZmFwU1NxZFptYXJaN0tsYjBzZmIxUksKU2tkMkR5YVB2Q01BQkZoZ2EvRlJSd3pIZGxCL3kxMHEvcUtGNm85VXBKMjdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWlRNkxlekFRbERGSHF3SndxVHpFClpnNGxzTTh3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQUlEUjlwZmZYcWFqd0VXd3U2cWpYVkFmNkNvVGZaRXEKa0NUN1dMOXZ1NjErQWlBOHhFTFVxSXNHSXc1eTlQM21rRnVHdDQzNGJDYkhraDF6OHJQT3RsZ2tDUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K - token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrZzJObEYyY0RKVlZGUnlOVVI1VFRJM04wazRORzFhV1ZSM2IwMTVTbnBSU2psTE1UQXpTa2RJUkdNaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpyZFdKbGMyeHBZMlV0WkdWdGJ5MWthWE4wY21saWRYUmxaQzF0WVhKcFlXUmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbXQxWW1WemJHbGpaUzF5WW1GakxYZHZjbXRsY2kxa1pXMXZMV052Ym5SeWIyeHNaWElpTENKcmRXSmxjbTVsZEdWekxtbHZMM05sY25acFkyVmhZMk52ZFc1MEwzTmxjblpwWTJVdFlXTmpiM1Z1ZEM1dVlXMWxJam9pYTNWaVpYTnNhV05sTFhKaVlXTXRkMjl5YTJWeUxXUmxiVzh0WTI5dWRISnZiR3hsY2lJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExuVnBaQ0k2SWpJeE9UTmtNamd4TFRRMVl6Z3RORGcxTlMwNU56QXdMVEF4T1RJM04yRmhaakpqWlNJc0luTjFZaUk2SW5ONWMzUmxiVHB6WlhKMmFXTmxZV05qYjNWdWREcHJkV0psYzJ4cFkyVXRaR1Z0Ynkxa2FYTjBjbWxpZFhSbFpDMXRZWEpwWVdSaU9tdDFZbVZ6YkdsalpTMXlZbUZqTFhkdmNtdGxjaTFrWlcxdkxXTnZiblJ5YjJ4c1pYSWlmUS5ZU1c3Sjl1N2NtUjVBTWhmMHY1ZVFtdmVpTUh5VlVJelBfTXBqV1NfY0pwcG0yWXJUOWZSNHRqejR4MGx3OHlSR2hrb1JuUGJXLXVMU1pDOExWNm0zX2Zxa0dHN3l3MFhQM3hCOWJsOEdxaGNVaG1rd0JCWHEtbEEybkZ6T0MtVTRqMEhOMnlRS0EtdzJQRUE4dnFGUlByUWVLckJwN0pLRXFUOFExbHMtTUNiUWNYZHJ0UDVNN255QXhSTHVsNnZJRlFkM0cwZU1RMzMwU3JNVlVsV2FaZ0NSbmJHZ2FGbnlwdks2RnlLZG9XUzg2ZzR6Sk1hZ0NRY2N3QnNibEN2anJEUHR4X1h6cVo2RWwwblpYanZTTFEyOGJGdU5DdmJ1QlI1T1JGSzI2aVZyQ3MxNnJYUlpSM2NTQXh3MTN0NDRGQVBraWg3ZlRJUEV6bjhnN0RTV0E= + namespace: + endpoint: + ca.crt: + token: cluster: name: demo-controller - endpoint: https://10.2.0.56:6443 + endpoint: https://:6443 netop: - networkInterface: enp1s0 + networkInterface: diff --git a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml index 9ff1a6eca4..ffd8327a7b 100644 --- a/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml +++ b/docs/guides/mariadb/distributed/overview/yamls/sliceoperator-worker.yaml @@ -1,11 +1,11 @@ ## Base64 encoded secret values from controller cluster controllerSecret: - namespace: a3ViZXNsaWNlLWRlbW8tZGlzdHJpYnV0ZWQtbWFyaWFkYg== - endpoint: aHR0cHM6Ly8xMC4yLjAuNTY6NjQ0Mw== - ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3SGhjTk1qVXdPREEyTURVeE5ERXhXaGNOTXpVd09EQTBNRFV4TkRFeApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUzTlRRME5UY3lOVEV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRZ0d0VVc3bFA5aWZLajNzN01rZmFwU1NxZFptYXJaN0tsYjBzZmIxUksKU2tkMkR5YVB2Q01BQkZoZ2EvRlJSd3pIZGxCL3kxMHEvcUtGNm85VXBKMjdvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVWlRNkxlekFRbERGSHF3SndxVHpFClpnNGxzTTh3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQUlEUjlwZmZYcWFqd0VXd3U2cWpYVkFmNkNvVGZaRXEKa0NUN1dMOXZ1NjErQWlBOHhFTFVxSXNHSXc1eTlQM21rRnVHdDQzNGJDYkhraDF6OHJQT3RsZ2tDUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K - token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklrZzJObEYyY0RKVlZGUnlOVVI1VFRJM04wazRORzFhV1ZSM2IwMTVTbnBSU2psTE1UQXpTa2RJUkdNaWZRLmV5SnBjM01pT2lKcmRXSmxjbTVsZEdWekwzTmxjblpwWTJWaFkyTnZkVzUwSWl3aWEzVmlaWEp1WlhSbGN5NXBieTl6WlhKMmFXTmxZV05qYjNWdWRDOXVZVzFsYzNCaFkyVWlPaUpyZFdKbGMyeHBZMlV0WkdWdGJ5MWthWE4wY21saWRYUmxaQzF0WVhKcFlXUmlJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5elpXTnlaWFF1Ym1GdFpTSTZJbXQxWW1WemJHbGpaUzF5WW1GakxYZHZjbXRsY2kxa1pXMXZMWGR2Y210bGNpSXNJbXQxWW1WeWJtVjBaWE11YVc4dmMyVnlkbWxqWldGalkyOTFiblF2YzJWeWRtbGpaUzFoWTJOdmRXNTBMbTVoYldVaU9pSnJkV0psYzJ4cFkyVXRjbUpoWXkxM2IzSnJaWEl0WkdWdGJ5MTNiM0pyWlhJaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sY25acFkyVXRZV05qYjNWdWRDNTFhV1FpT2lJMU9URmxNR0UyWlMwd01EWmpMVFJrWkdZdFlqVmhNQzA0TkRsbVpqQTBOalkzTTJFaUxDSnpkV0lpT2lKemVYTjBaVzA2YzJWeWRtbGpaV0ZqWTI5MWJuUTZhM1ZpWlhOc2FXTmxMV1JsYlc4dFpHbHpkSEpwWW5WMFpXUXRiV0Z5YVdGa1lqcHJkV0psYzJ4cFkyVXRjbUpoWXkxM2IzSnJaWEl0WkdWdGJ5MTNiM0pyWlhJaWZRLkhmaE5Ba1J5VmZCQ0pVV1lLTDN3MXhRdVVNakJzMUZFLVotelNRX3pPNTU3NWttUkczUzhyMUlZV1FJdzNOLWZydzhvUlpTRURwWWhsdnhVVTlxYUFsXzJuZkczVDE5OUZZc2hMNEt4U0JWZlhaT0puaGdaS1ZuOW40MTY2VHAwV0NRVTZ1WE1MZE80TTgwV21neGVORWRzVUtYU05iekVHRWRFY3oteWg3dkRteThQUmwyVFZjUFFTamJkQ0l0UWxBMTQ0STVraWMxSGRCNVdiTHR1WDZ0N2hIaHNOYzlNZkYwZXBBZkd4a0YwVDFTWHdNcDBHY3c5cW52OVEzSk91d2liemdXTGF0aUIyWHQtVEdnNWtSZTlCMzduelBlQi1uTmtRY3FGeWEzb2x2Q01QNDRMcW9CbGdqMTJlRGhjOFB6dEFseXRiQ3Z1S1l3Y0lsYk9PUQ== + namespace: + endpoint: + ca.crt: + token: cluster: name: demo-worker - endpoint: https://10.2.0.60:6443 + endpoint: https://:6443 netop: - networkInterface: enp1s0 + networkInterface: diff --git a/docs/guides/postgres/distributed/overview/index.md b/docs/guides/postgres/distributed/overview/index.md index 3775d2970d..458e77a33b 100644 --- a/docs/guides/postgres/distributed/overview/index.md +++ b/docs/guides/postgres/distributed/overview/index.md @@ -9,11 +9,12 @@ menu: menu_name: docs_{{ .version }} section_menu_id: guides --- + # Distributed Postgres Cluster Overview ## Introduction -KubeDB enables distributed Postgres deployments across multiple Kubernetes clusters, providing a scalable, highly available, and resilient database solution. By integrating **Open Cluster Management (OCM)** for multi-cluster orchestration and **KubeSlice** for seamless pod-to-pod network connectivity, KubeDB simplifies the deployment and management of Postgres instances across clusters. A **PodPlacementPolicy** ensures precise control over pod scheduling, allowing you to distribute Postgres pods across clusters for optimal resource utilization and fault tolerance. +KubeDB enables distributed Postgres deployments across multiple Kubernetes clusters, providing a scalable, highly available, and resilient database solution. By integrating **Open Cluster Management (OCM)** for multi-cluster orchestration and **KubeSlice** for seamless pod-to-pod network connectivity, KubeDB simplifies the deployment and management of Postgres instances across clusters. A **PlacementPolicy** ensures precise control over pod scheduling, allowing you to distribute Postgres pods across clusters for optimal resource utilization and fault tolerance. This guide provides a step-by-step process to deploy a distributed Postgres cluster, including prerequisites, configuration, and verification steps. It assumes familiarity with Kubernetes and basic database concepts. @@ -21,23 +22,23 @@ This guide provides a step-by-step process to deploy a distributed Postgres clus ## Understanding OCM Hub and Spoke Clusters -In an **Open Cluster Management (OCM)** setup, clusters are categorized as: +In an **Open Cluster Management (OCM)** setup, clusters are categorized as follows: -* **Hub Cluster**: The central control plane where policies, [applications](), and resources are defined and managed. It orchestrates the lifecycle of applications deployed across spoke clusters. -* **Spoke Cluster**: Managed clusters registered with the hub, running the actual workloads (e.g., Postgres pods). +- **Hub Cluster**: The central control plane where policies, applications, and resources are defined and managed. It orchestrates the lifecycle of applications deployed across spoke clusters. +- **Spoke Cluster**: Managed clusters registered with the hub that run the actual workloads (e.g., Postgres pods). -When a spoke cluster (e.g., `demo-worker`) is joined to the hub using the `clusteradm join` command, OCM creates a namespace on the hub cluster matching the spoke cluster's name (e.g., `demo-worker`). This namespace is used to manage resources specific to the spoke cluster from the hub. +When a spoke cluster (e.g., `demo-worker`) is joined to the hub using the `clusteradm join` command, OCM creates a namespace on the hub cluster that matches the spoke cluster's name (e.g., `demo-worker`). This namespace is used to manage resources specific to the spoke cluster from the hub. ## Prerequisites -Before deploying a distributed Postgres cluster, ensure the following: +Before deploying a distributed Postgres cluster, ensure the following requirements are met: - **Kubernetes Clusters**: Multiple Kubernetes clusters (version 1.26 or higher) configured and accessible. -- **Node Requirements**: Each Kubernetes node should have at least 4 vCPUs and 16GB of RAM. -- **Open Cluster Management (OCM)**: Install `clusteradm` as per the [OCM Quick Start Guide](https://open-cluster-management.io/docs/getting-started/quick-start/). +- **Node Requirements**: Each Kubernetes node should have at least 4 vCPUs and 16 GB of RAM. +- **Open Cluster Management (OCM)**: Install `clusteradm` as described in the [OCM Quick Start Guide](https://open-cluster-management.io/docs/getting-started/quick-start/). - **kubectl**: Installed and configured to interact with all clusters. - **Helm**: Installed for deploying the KubeDB Operator and KubeSlice components. -- **Persistent Storage**: A storage class (e.g., `local-path` or cloud provider-specific) configured for persistent volumes. +- **Persistent Storage**: A storage class (e.g., `local-path` or a cloud provider-specific option) configured for persistent volumes. ## Configuration Steps @@ -45,164 +46,197 @@ Follow these steps to deploy a distributed Postgres cluster across multiple Kube ### Step 1: Set Up Open Cluster Management (OCM) -1. **Configure KUBECONFIG**: - Ensure your `KUBECONFIG` is set up to switch between clusters. This guide uses two clusters: `demo-controller` (hub and spoke) and `demo-worker` (spoke). - - ```bash - kubectl config get-contexts - ``` - - **Output**: - ``` - CURRENT NAME CLUSTER AUTHINFO NAMESPACE - * demo-controller demo-controller demo-controller - demo-worker demo-worker demo-worker - ``` - -2. **Initialize the OCM Hub**: - On the `demo-controller` cluster, initialize the OCM hub. - - ```bash - kubectl config use-context demo-controller - clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true - ``` - -3. **Verify Hub Deployment**: - Check the pods in the `open-cluster-management-hub` namespace to ensure all components are running. - - ```bash - kubectl get pods -n open-cluster-management-hub - ``` - - **Output**: - ``` - NAME READY STATUS RESTARTS AGE - cluster-manager-addon-manager-controller-5f99f56896-qpzj8 1/1 Running 0 7m2s - cluster-manager-placement-controller-597d5ff644-wqjq2 1/1 Running 0 7m2s - cluster-manager-registration-controller-6d79d7dcc6-b8h9p 1/1 Running 0 7m2s - cluster-manager-registration-webhook-5d88cf97c7-2sq5m 1/1 Running 0 7m2s - cluster-manager-work-controller-7468bf4dc-5qn6q 1/1 Running 0 7m2s - cluster-manager-work-webhook-c5875947-d272b 1/1 Running 0 7m2s - ``` - - All pods should be in the `Running` state with `1/1` readiness and no restarts, indicating a successful hub deployment. - -4. **Register Spoke Cluster (`demo-worker`)**: - Obtain the join token from the hub cluster. - - ```bash - clusteradm get token - ``` - - **Output**: - ``` - token= - please log on spoke and run: - clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name - ``` - - On the `demo-worker` cluster, join it to the hub, including the `RawFeedbackJsonString` feature gate for resource feedback. - - ```bash - kubectl config use-context demo-worker - clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true - ``` - -5. **Accept Spoke Cluster**: - On the `demo-controller` cluster, accept the `demo-worker` cluster. - - ```bash - kubectl config use-context demo-controller - clusteradm accept --clusters demo-worker - ``` - - **Note**: It may take a few attempts (e.g., retry every 10 seconds) if the cluster is not immediately available. - - **Output** (on success): - ``` - Starting approve csrs for the cluster demo-worker - CSR demo-worker-2p2pb approved - set hubAcceptsClient to true for managed cluster demo-worker - Your managed cluster demo-worker has joined the Hub successfully. - ``` - -6. **Verify Namespace Creation**: - Confirm that a namespace for `demo-worker` was created on the hub cluster. - - ```bash - kubectl get ns - ``` - - **Output**: - ``` - NAME STATUS AGE - default Active 99m - demo-worker Active 58s - kube-node-lease Active 99m - kube-public Active 99m - kube-system Active 99m - open-cluster-management Active 6m7s - open-cluster-management-hub Active 5m32s - ``` - -7. **Register `demo-controller` as a Spoke Cluster**: - Repeat the join and accept process for `demo-controller` to also act as a spoke cluster. - - ```bash - kubectl config use-context demo-controller - clusteradm join --hub-token --hub-apiserver https://10.2.0.56:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true - clusteradm accept --clusters demo-controller - ``` - - Verify the namespace for `demo-controller`. - - ```bash - kubectl get ns - ``` - - **Output**: - ``` - NAME STATUS AGE - default Active 104m - demo-controller Active 3s - demo-worker Active 6m7s - kube-node-lease Active 104m - kube-public Active 104m - kube-system Active 104m - open-cluster-management Active 11m - open-cluster-management-agent Active 37s - open-cluster-management-agent-addon Active 34s - open-cluster-management-hub Active 10m - ``` - -### Step 2: Configure OCM WorkConfiguration (Optional) - -If you did not follow the provided OCM installation steps, update the `klusterlet` resource to enable feedback retrieval. - -```bash -kubectl edit klusterlet klusterlet -``` +#### 1. Configure KUBECONFIG -Add the following under the `spec` field: +Ensure your `KUBECONFIG` is set up to switch between clusters. This guide uses two clusters: `demo-controller` (hub and spoke) and `demo-worker` (spoke). + +```bash +$ kubectl config get-contexts +``` + +**Output:** + +```bash +CURRENT NAME CLUSTER AUTHINFO NAMESPACE +* demo-controller demo-controller demo-controller + demo-worker demo-worker demo-worker +``` + +#### 2. Initialize the OCM Hub + +On the `demo-controller` cluster, initialize the OCM hub: + +```bash +$ kubectl config use-context demo-controller +$ clusteradm init --wait --feature-gates=ManifestWorkReplicaSet=true +``` + +#### 3. Verify Hub Deployment + +Check the pods in the `open-cluster-management-hub` namespace to ensure all components are running: + +```bash +$ kubectl get pods -n open-cluster-management-hub +``` + +**Output:** -```yaml -workConfiguration: - featureGates: - - feature: RawFeedbackJsonString - mode: Enable - hubKubeAPIBurst: 100 - hubKubeAPIQPS: 50 - kubeAPIBurst: 100 - kubeAPIQPS: 50 +``` +NAME READY STATUS RESTARTS AGE +cluster-manager-addon-manager-controller-5f99f56896-qpzj8 1/1 Running 0 7m2s +cluster-manager-placement-controller-597d5ff644-wqjq2 1/1 Running 0 7m2s +cluster-manager-registration-controller-6d79d7dcc6-b8h9p 1/1 Running 0 7m2s +cluster-manager-registration-webhook-5d88cf97c7-2sq5m 1/1 Running 0 7m2s +cluster-manager-work-controller-7468bf4dc-5qn6q 1/1 Running 0 7m2s +cluster-manager-work-webhook-c5875947-d272b 1/1 Running 0 7m2s +``` + +All pods should be in the `Running` state with `1/1` readiness and no restarts, indicating a successful hub deployment. + +#### 4. Register Spoke Cluster (`demo-worker`) + +Obtain the join token from the hub cluster: + +```bash +$ clusteradm get token +``` + +**Output:** + +``` +token= +please log on spoke and run: +clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name +``` + +On the `demo-worker` cluster, join it to the hub. Include the `RawFeedbackJsonString` feature gate for resource feedback: + +```bash +$ kubectl config use-context demo-worker +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-worker --feature-gates=RawFeedbackJsonString=true +``` + +#### 5. Accept Spoke Cluster + +On the `demo-controller` cluster, accept the `demo-worker` cluster: + +```bash +$ kubectl config use-context demo-controller +$ clusteradm accept --clusters demo-worker +``` + +> **Note:** It may take a few attempts (e.g., retry every 10 seconds) if the cluster is not immediately available. + +**Output (on success):** + + +- Starting approve csrs for the cluster demo-worker +- CSR demo-worker-2p2pb approved +- set hubAcceptsClient to true for managed cluster demo-worker +- Your managed cluster demo-worker has joined the Hub successfully. + + +#### 6. Verify Namespace Creation + +Confirm that a namespace for `demo-worker` was created on the hub cluster: + +```bash +$ kubectl get ns +``` + +**Output:** + +```bash +NAME STATUS AGE +default Active 99m +demo-worker Active 58s +kube-node-lease Active 99m +kube-public Active 99m +kube-system Active 99m +open-cluster-management Active 6m7s +open-cluster-management-hub Active 5m32s +``` + +#### 7. Register `demo-controller` as a Spoke Cluster + +Repeat the join and accept process for `demo-controller` so it can also act as a spoke cluster: + +```bash +$ kubectl config use-context demo-controller +$ clusteradm join --hub-token --hub-apiserver https://:6443 --cluster-name demo-controller --feature-gates=RawFeedbackJsonString=true +$ clusteradm accept --clusters demo-controller +``` + +Verify the namespace for `demo-controller`: + +```bash +$ kubectl get ns +``` + +**Output:** + +``` +NAME STATUS AGE +default Active 104m +demo-controller Active 3s +demo-worker Active 6m7s +kube-node-lease Active 104m +kube-public Active 104m +kube-system Active 104m +open-cluster-management Active 11m +open-cluster-management-agent Active 37s +open-cluster-management-agent-addon Active 34s +open-cluster-management-hub Active 10m +``` + +#### 8. Verify OCM Roles + +After registration, use these commands to confirm which cluster is the hub and which are spokes: + +```bash +# Hub: lists all registered spoke clusters +$ kubectl get managedclusters + +# Spoke: shows this cluster's registered name +$ kubectl get klusterlet klusterlet -o jsonpath='{.spec.clusterName}' + +# Hub components run only on the hub cluster +$ kubectl get pods -n open-cluster-management-hub + +# Spoke agent runs on every spoke cluster +$ kubectl get pods -n open-cluster-management-agent +``` + +### Step 2: Configure OCM WorkConfiguration + +Run this on **every spoke cluster** (`demo-controller` and `demo-worker`). This enables the `RawFeedbackJsonString` feature gate that KubeDB requires to read pod status across clusters, and raises the API rate limits to prevent throttling. + +> **Note:** Even if you passed `--feature-gates=RawFeedbackJsonString=true` during `clusteradm join`, the rate limit fields are not set by that flag. Run this patch on all spokes regardless. +> +> **Why this matters:** KubeDB uses OCM's ManifestWork feedback mechanism to watch the status of Postgres pods on remote spoke clusters. Without `RawFeedbackJsonString`, the KubeDB provisioner on the hub never receives pod status updates from spokes and the distributed Postgres CR will stay in a non-Ready state indefinitely. The rate limits prevent the klusterlet agent from being API-throttled during initial cluster formation. + +```bash +$ kubectl patch klusterlet klusterlet --type=merge -p '{ + "spec": { + "workConfiguration": { + "featureGates": [{"feature": "RawFeedbackJsonString", "mode": "Enable"}], + "hubKubeAPIBurst": 100, + "hubKubeAPIQPS": 50, + "kubeAPIBurst": 100, + "kubeAPIQPS": 50 + } + } +}' ``` Verify the configuration: ```bash -kubectl get klusterlet klusterlet -oyaml +$ kubectl get klusterlet klusterlet -oyaml ``` -**Sample Output** (abridged): +**Sample Output (abridged):** + ```yaml apiVersion: operator.open-cluster-management.io/v1 kind: Klusterlet @@ -220,290 +254,235 @@ spec: kubeAPIQPS: 50 ``` - ### Step 3: Configure KubeSlice for Network Connectivity KubeSlice enables pod-to-pod communication across clusters. Install the KubeSlice Controller on the `demo-controller` cluster and the KubeSlice Worker on both `demo-controller` and `demo-worker` clusters. -1. **Install KubeSlice Controller**: - On `demo-controller`, create a `controller.yaml` file: - - ```yaml - kubeslice: - controller: - loglevel: info - rbacResourcePrefix: kubeslice-rbac - projectnsPrefix: kubeslice - endpoint: https://10.2.0.56:6443 - ``` - - Deploy the controller using Helm: - - ```bash - helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ - --version v2025.7.31 \ - -f controller.yaml \ - --namespace kubeslice-controller \ - --create-namespace \ - --wait --burst-limit=10000 --debug - ``` - - Verify the installation: - - ```bash - kubectl get pods -n kubeslice-controller - ``` - - **Output**: - ``` - NAME READY STATUS RESTARTS AGE - kubeslice-controller-manager-7fd756fff6-5kddd 2/2 Running 0 98s - ``` - -2. **Create a KubeSlice Project**: - Create a `project.yaml` file: - - ```yaml - apiVersion: controller.kubeslice.io/v1alpha1 - kind: Project - metadata: - name: demo-distributed-postgres - namespace: kubeslice-controller - spec: - serviceAccount: - readWrite: - - admin - ``` - - Apply the project: - - ```bash - kubectl apply -f project.yaml - ``` - - Verify: - - ```bash - kubectl get project -n kubeslice-controller - ``` - - **Output**: - ``` - NAME AGE - demo-distributed-postgres 31s - ``` - - Check service accounts: - - ```bash - kubectl get sa -n kubeslice-demo-distributed-postgres - ``` - - **Output**: - ``` - NAME SECRETS AGE - default 0 69s - kubeslice-rbac-rw-admin 1 68s - ``` - -3. **Label Nodes for KubeSlice**: - Assign the `kubeslice.io/node-type=gateway` label to node(where worker operator will deploy) in both clusters. - - On `demo-controller`: - - ```bash - kubectl get nodes - kubectl label node demo-master kubeslice.io/node-type=gateway - ``` - - On `demo-worker`: - - ```bash - kubectl config use-context demo-worker - kubectl get nodes - kubectl label node demo-worker kubeslice.io/node-type=gateway - ``` - -4. **Register Clusters with KubeSlice**: - Identify the network interface for both clusters by running the command on the node. - - ```bash - ip route get 8.8.8.8 | awk '{ print $5 }' - ``` - - **Output** (example): - ``` - enp1s0 - ``` - - Create a `registration.yaml` file: - - ```yaml - apiVersion: controller.kubeslice.io/v1alpha1 - kind: Cluster - metadata: - name: demo-controller - namespace: kubeslice-demo-distributed-postgres - spec: - networkInterface: enp1s0 - clusterProperty: {} - --- - apiVersion: controller.kubeslice.io/v1alpha1 - kind: Cluster - metadata: - name: demo-worker - namespace: kubeslice-demo-distributed-postgres - spec: - networkInterface: enp1s0 - clusterProperty: {} - ``` - - Apply on `demo-controller`: - - ```bash - kubectl apply -f registration.yaml - ``` - - Verify: - - ```bash - kubectl get clusters -n kubeslice-demo-distributed-postgres - ``` - - **Output**: - ``` - NAME AGE - demo-controller 9s - demo-worker 9s - ``` - -5. **Register KubeSlice Worker Clusters**: - Create a `secrets.sh` script to generate worker configuration: - -```bash - # The script returns a kubeconfig for the service account given -# you need to have kubectl on PATH with the context set to the cluster you want to create the config for -# Cosmetics for the created config -firstWorkerSecretName=$1 -# cluster name what you given in clusters registration -clusterName=$2 -# the Namespace and ServiceAccount name that is used for the config -namespace=$3 -# Need to give correct network interface value like ens160, eth0 etc -networkInterface=$4 -# kubectl cluster-info of respective worker-cluster -worker_endpoint=$5 -###################### -# actual script starts -set -o errexit - -### Fetch Worker cluster Secrets ### -PROJECT_NAMESPACE=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath={.data.namespace}) -CONTROLLER_ENDPOINT=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath={.data.controllerEndpoint}) -CA_CRT=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath='{.data.ca\.crt}') -TOKEN=$(kubectl get secrets $firstWorkerSecretName -n $namespace -o jsonpath={.data.token}) - -echo " +#### 1. Install KubeSlice Controller + +On `demo-controller`, get the hub API server address first: + +```bash +$ kubectl cluster-info | grep 'Kubernetes control plane' +``` + +Use the IP and port from that output as the `endpoint` value. Create a `controller.yaml` file: + +```yaml +kubeslice: + controller: + loglevel: info + rbacResourcePrefix: kubeslice-rbac + projectnsPrefix: kubeslice + endpoint: https://:6443 +``` + +Deploy the controller using Helm: + +```bash +$ helm upgrade -i kubeslice-controller oci://ghcr.io/appscode-charts/kubeslice-controller \ + --version v2026.1.15 \ + -f controller.yaml \ + --namespace kubeslice-controller \ + --create-namespace \ + --set ocm.enabled=true \ + --wait --burst-limit=10000 --debug +``` + +Verify the installation: + +```bash +$ kubectl get pods -n kubeslice-controller +``` + +**Output:** + +``` +NAME READY STATUS RESTARTS AGE +kubeslice-controller-manager-7fd756fff6-5kddd 2/2 Running 0 98s +``` + +#### 2. Create a KubeSlice Project + +Create a `project.yaml` file: + +```yaml +apiVersion: controller.kubeslice.io/v1alpha1 +kind: Project +metadata: + name: demo-distributed-postgres + namespace: kubeslice-controller +spec: + serviceAccount: + readWrite: + - admin +``` + +Apply the project: + +```bash +$ kubectl apply -f project.yaml +``` + +Verify: + +```bash +$ kubectl get project -n kubeslice-controller +``` + +**Output:** + +```bash +NAME AGE +demo-distributed-postgres 31s +``` + +Check service accounts: + +```bash +$ kubectl get sa -n kubeslice-demo-distributed-postgres +``` + +**Output:** + +``` +NAME SECRETS AGE +default 0 69s +kubeslice-rbac-rw-admin 1 68s +``` + +#### 3. Label Nodes for KubeSlice + +Assign the `kubeslice.io/node-type=gateway` label to the node where the worker operator will be deployed in both clusters. + +On `demo-controller`: + +```bash +$ kubectl get nodes +$ kubectl label node kubeslice.io/node-type=gateway +``` + +On `demo-worker`: + +```bash +$ kubectl config use-context demo-worker +$ kubectl get nodes +$ kubectl label node kubeslice.io/node-type=gateway +``` + +#### 4. Register Clusters with KubeSlice + +Identify the network interface for each cluster by running the following command **on the gateway node of each cluster**: + +```bash +$ ip route get 8.8.8.8 | awk '{ print $5 }' +``` + +**Output (example):** + +``` +enp1s0 +``` + +> **Important:** The `networkInterface` value must match the primary network interface of each cluster's gateway node. Run the command above on **each cluster separately** and use that cluster's output as its `networkInterface` value. Each cluster may have a **different interface name** — do not assume they are the same. Using the wrong interface name will cause the WireGuard gateway to silently fail and cross-cluster Postgres replication will never connect. +> +> Example: if `demo-controller` returns `enp3s0` and `demo-worker` returns `eth0`, use those exact values in the YAML below. + +Create a `registration.yaml` file: + +> **Note:** +> - The cluster name must exactly match the name of the OCM (spoke) cluster. +> - The corresponding `ManagedClusterAddOn` resource must be created in the namespace that bears the same name as the cluster to set up the KubeSlice worker automatically. + +```yaml +apiVersion: controller.kubeslice.io/v1alpha1 +kind: Cluster +metadata: + name: demo-controller + namespace: kubeslice-demo-distributed-postgres +spec: + networkInterface: # replace with output of ip route command + clusterProperty: {} +--- +apiVersion: controller.kubeslice.io/v1alpha1 +kind: Cluster +metadata: + name: demo-worker + namespace: kubeslice-demo-distributed-postgres +spec: + networkInterface: # replace with output of ip route command + clusterProperty: {} --- -## Base64 encoded secret values from controller cluster -controllerSecret: -namespace: ${PROJECT_NAMESPACE} -endpoint: ${CONTROLLER_ENDPOINT} -ca.crt: ${CA_CRT} -token: ${TOKEN} -cluster: -name: ${clusterName} -endpoint: ${worker_endpoint} -netop: -networkInterface: ${networkInterface} -" -``` -Command to generate the worker configuration: -```bash -sh secrets.sh -``` - -Get secrets for the project namespace: - - ```bash - kubectl get secrets -n kubeslice-demo-distributed-postgres - ``` - -**Output**: - ``` - NAME TYPE DATA AGE - kubeslice-rbac-rw-admin kubernetes.io/service-account-token 3 17m - kubeslice-rbac-worker-demo-controller kubernetes.io/service-account-token 5 5m8s - kubeslice-rbac-worker-demo-worker kubernetes.io/service-account-token 5 5m8s - ``` - -Get the `demo-worker` cluster endpoint: - - ```bash - kubectl cluster-info --context demo-worker --kubeconfig $HOME/.kube/config - ``` - -**Output**: - ``` - Kubernetes control plane is running at https://10.2.0.60:6443 - ``` - -Run the script for `demo-worker`: - - ```bash - sh secrets.sh kubeslice-rbac-worker-demo-worker demo-worker kubeslice-demo-distributed-postgres enp1s0 https://10.2.0.60:6443 > sliceoperator-worker.yaml - ``` - -Install the KubeSlice worker on `demo-worker`: - - ```bash - kubectl config use-context demo-worker - helm upgrade -i kubeslice-worker oci://ghcr.io/appscode-charts/kubeslice-worker \ - --version v2025.7.31 \ - -f sliceoperator-worker.yaml \ - --namespace kubeslice-system \ - --create-namespace \ - --wait --burst-limit=10000 --debug - ``` - -Repeat for `demo-controller`: - - ```bash - kubectl config use-context demo-controller - sh secrets.sh kubeslice-rbac-worker-demo-controller demo-controller kubeslice-demo-distributed-postgres enp1s0 https://10.2.0.56:6443 > sliceoperator-controller.yaml - helm upgrade -i kubeslice-worker oci://ghcr.io/appscode-charts/kubeslice-worker \ - --version v2025.7.31 \ - -f sliceoperator-controller.yaml \ - --namespace kubeslice-system \ - --create-namespace \ - --wait --burst-limit=10000 --debug - ``` - -Verify the worker installation: - - ```bash - kubectl get pods -n kubeslice-system - ``` - -**Output**: - ``` - NAME READY STATUS RESTARTS AGE - forwarder-kernel-bw5l4 1/1 Running 0 4m43s - kubeslice-dns-6bd9749f4d-pvh7g 1/1 Running 0 4m43s - kubeslice-install-crds-szhvc 0/1 Completed 0 4m56s - kubeslice-netop-g4dfn 1/1 Running 0 4m43s - kubeslice-operator-949b7d6f7-9wj7h 2/2 Running 0 4m43s - kubeslice-postdelete-job-ctlzt 0/1 Completed 0 20m - nsm-delete-webhooks-ndksl 0/1 Completed 0 20m - nsm-install-crds-5z4j9 0/1 Completed 0 4m53s - nsmgr-zzwgh 2/2 Running 0 4m43s - registry-k8s-979455d6d-q2j8x 1/1 Running 0 4m43s - spire-install-clusterid-cr-qwqlr 0/1 Completed 0 4m47s - spire-install-crds-cnbjh 0/1 Completed 0 4m50s - ``` - - **Onboard Database Namespace** +apiVersion: addon.open-cluster-management.io/v1alpha1 +kind: ManagedClusterAddOn +metadata: + name: kubeslice + namespace: demo-controller +spec: + installNamespace: kubeslice-system + configs: + - name: demo-controller + namespace: kubeslice-demo-distributed-postgres + group: controller.kubeslice.io + resource: clusters +--- +apiVersion: addon.open-cluster-management.io/v1alpha1 +kind: ManagedClusterAddOn +metadata: + name: kubeslice + namespace: demo-worker +spec: + installNamespace: kubeslice-system + configs: + - name: demo-worker + namespace: kubeslice-demo-distributed-postgres + group: controller.kubeslice.io + resource: clusters +``` +Apply on `demo-controller`: + +```bash +$ kubectl apply -f registration.yaml +``` + +Verify OCM is deploying the KubeSlice worker manifests to each cluster: + +```bash +$ kubectl get managedclusteraddon -A +``` + +**Output:** + +``` +NAMESPACE NAME AVAILABLE DEGRADED PROGRESSING +demo-controller kubeslice Unknown True +demo-worker kubeslice Unknown True +``` + +`PROGRESSING: True` means OCM is actively deploying. Wait until `kubeslice-operator` shows `2/2 Running` on both clusters before proceeding: + +```bash +# Run on each spoke cluster +$ kubectl get pods -n kubeslice-system --watch +``` + +**Expected output (after KubeSlice worker is fully deployed):** + +``` +NAME READY STATUS RESTARTS AGE +forwarder-kernel-bw5l4 1/1 Running 0 4m43s +kubeslice-dns-6bd9749f4d-pvh7g 1/1 Running 0 4m43s +kubeslice-install-crds-szhvc 0/1 Completed 0 4m56s +kubeslice-netop-g4dfn 1/1 Running 0 4m43s +kubeslice-operator-949b7d6f7-9wj7h 2/2 Running 0 4m43s +nsc-grpc-server-sbjj7 1/1 Running 0 4m43s +nsm-install-crds-5z4j9 0/1 Completed 0 4m53s +nsmgr-zzwgh 2/2 Running 0 4m43s +registry-k8s-979455d6d-q2j8x 1/1 Running 0 4m43s +``` + +#### 5. Onboard Application Namespace Create a `SliceConfig` to onboard the `demo` (application) and `kubedb` (operator) namespaces for network connectivity. Create a `sliceconfig.yaml` file: @@ -552,75 +531,200 @@ spec: Apply the `SliceConfig`: ```bash -kubectl apply -f sliceconfig.yaml +$ kubectl apply -f sliceconfig.yaml +``` + +After the SliceConfig is applied, a `vl3-slice-router` pod will appear in `kubeslice-system` on each cluster, indicating the slice VPN tunnel is being established. + +#### 6. Configure DNS for KubeSlice + +Update CoreDNS to forward `*.slice.local` traffic to the KubeSlice DNS service. Run the following steps on **every cluster** in the slice. + +> **Important:** CoreDNS must be updated and restarted on all clusters before proceeding to Step 4 (KubeDB install). Postgres nodes use `.slice.local` DNS names to discover each other across clusters. If DNS is not configured before Postgres pods start, replication will not form. + +Get the KubeSlice DNS service IP address on each cluster: + +```bash +$ kubectl get svc -n kubeslice-system -owide -l 'app=kubeslice-dns' ``` +**Output:** + +```bash +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR +kubeslice-dns ClusterIP 10.43.172.191 53/UDP,53/TCP 8d app=kubeslice-dns +``` + +Add the following block to the **top** of your CoreDNS `Corefile` ConfigMap (replace the IP with the one from your cluster): + +``` +slice.local:53 { + errors + cache 30 + forward . 10.43.172.191 +} +``` + +Example of the full CoreDNS ConfigMap after editing: + +```bash +$ kubectl get cm -n kube-system coredns -oyaml +``` + +**Output:** + +```yaml +apiVersion: v1 +data: + Corefile: | + slice.local:53 { + errors + cache 30 + forward . 10.43.172.191 + } + .:53 { + errors + health + ready + kubernetes cluster.local in-addr.arpa ip6.arpa { + pods insecure + fallthrough in-addr.arpa ip6.arpa + } + hosts /etc/coredns/NodeHosts { + ttl 60 + reload 15s + fallthrough + } + prometheus :9153 + cache 30 + loop + reload + loadbalance + import /etc/coredns/custom/*.override + forward . /etc/resolv.conf + } + import /etc/coredns/custom/*.server + NodeHosts: | + 10.2.0.248 demo-worker +kind: ConfigMap +metadata: + name: coredns + namespace: kube-system +``` + +After editing the ConfigMap, restart CoreDNS to apply the change: + +```bash +$ kubectl rollout restart deploy/coredns -n kube-system +``` + +Repeat the DNS configuration steps on every cluster in the slice. + ### Step 4: Install the KubeDB Operator -Install the KubeDB Operator on the `demo-controller` cluster to manage the Postgres instance. +> **Note:** Install the KubeDB operator only on the hub cluster (`demo-controller`). The operator manages Postgres pods on spoke clusters through OCM — no KubeDB installation is needed on `demo-worker`. #### Get a Free License -Download a FREE license from [AppsCode License Server](https://appscode.com/issue-license?p=kubedb). Get the license for `demo-controller` cluster. + +The KubeDB license is tied to the `kube-system` namespace UID of the hub cluster and has an expiry date. Get your cluster UID and verify the license before installing: ```bash -helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ - --version v2025.8.31 \ +# Get your cluster UID (required when requesting the license) +$ kubectl get ns kube-system -o jsonpath='{.metadata.uid}' + +# Verify the license is not expired +$ openssl x509 -noout -enddate -in $HOME/Downloads/kubedb-license-.txt +``` + +If expired or not yet obtained, download a FREE license from the [AppsCode License Server](https://appscode.com/issue-license?p=kubedb) using the cluster UID above. + +```bash +$ helm upgrade -i kubedb oci://ghcr.io/appscode-charts/kubedb \ + --version v2026.2.26 \ --namespace kubedb --create-namespace \ - --set-file global.license=$HOME/Downloads/kubedb-license-cd548cce-5141-4ed3-9276-6d9578707f12.txt \ + --set-file global.license=$HOME/Downloads/kubedb-license-.txt \ --set petset.features.ocm.enabled=true \ --wait --burst-limit=10000 --debug ``` -Note: `--set petset.features.ocm.enabled=true` must be set to enable Postgres Distributed feature. Also `--set-file global.license=` should point to your kubedb license file. -Follow the [KubeDB Installation Guide](https://kubedb.com/docs/v2025.6.30/setup/install/kubedb/) for additional details. +> **Note:** The `--set petset.features.ocm.enabled=true` flag must be set to enable the Postgres Distributed feature. + +For additional details, refer to the [KubeDB Installation Guide](https://kubedb.com/docs/v2026.4.27/setup/). + +Verify that the pods are running: + +```bash +$ kubectl get pods -n kubedb +``` + +**Output:** -### Step 5: Define a PodPlacementPolicy +```bash +NAME READY STATUS RESTARTS AGE +kubedb-kubedb-autoscaler-0 1/2 Running 0 44s +kubedb-kubedb-ops-manager-0 1/2 Running 0 43s +kubedb-kubedb-provisioner-0 1/2 Running 0 43s +kubedb-kubedb-webhook-server-df667cd85-tjdp9 2/2 Running 0 44s +kubedb-petset-cf9f5b6f4-d9558 2/2 Running 0 44s +kubedb-sidekick-5dbf7bcf64-4b8cw 2/2 Running 0 44s +``` -Create a `PlacementPolicy` to control pod distribution across clusters. Example: +### Step 5: Define a PlacementPolicy + +You can define the `storageClassName` under `spec.clusterSpreadConstraint.distributionRules` for each cluster. If not explicitly specified, the clusters will automatically select an appropriate storage class. Additionally, you can control replica placement by defining which replicas belong to which cluster. When scaling the number of replicas, the distribution must also be specified at the cluster level. In this example, we are using three replicas. + +To manage pod distribution across clusters, create a `PlacementPolicy`. For this purpose, define a `pod-placement-policy.yaml` file as shown below: ```yaml apiVersion: apps.k8s.appscode.com/v1 kind: PlacementPolicy metadata: - labels: - app.kubernetes.io/managed-by: Helm - name: distributed-postgres + labels: + app.kubernetes.io/managed-by: Helm + name: distributed-postgres spec: - clusterSpreadConstraint: - distributionRules: - - clusterName: demo-controller - replicaIndices: - - 0 - - 2 - - clusterName: demo-worker - replicaIndices: - - 1 - slice: - projectNamespace: kubeslice-demo-distributed-mariadb - sliceName: demo-slice - nodeSpreadConstraint: - maxSkew: 1 - whenUnsatisfiable: ScheduleAnyway - zoneSpreadConstraint: - maxSkew: 1 - whenUnsatisfiable: ScheduleAnyway - - + clusterSpreadConstraint: + distributionRules: + - clusterName: demo-controller + storageClassName: local-path # optional; omit to use the cluster's default storage class + replicaIndices: + - 0 + - 2 + - clusterName: demo-worker + storageClassName: local-path # optional; omit to use the cluster's default storage class + replicaIndices: + - 1 + slice: + projectNamespace: kubeslice-demo-distributed-postgres + sliceName: demo-slice + nodeSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway + zoneSpreadConstraint: + maxSkew: 1 + whenUnsatisfiable: ScheduleAnyway ``` This policy schedules: -- `postgres-0` and `postgres-2` on `demo-controller`. -- `postgres-1` on `demo-worker`. + +- `postgres-0` and `postgres-2` on `demo-controller` +- `postgres-1` on `demo-worker` Apply the policy on `demo-controller`: ```bash -kubectl apply -f podplacementpolicy.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f pod-placement-policy.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 6: Create a Distributed Postgres Instance -Define a Postgres custom resource with `spec.distributed: true` and reference the `PlacementPolicy`. Example: +Create the `demo` namespace first: + +```bash +$ kubectl create namespace demo +``` + +Define a Postgres custom resource with `spec.distributed` set to `true` and reference the `PlacementPolicy`. Create a `postgres.yaml` file: ```yaml apiVersion: kubedb.com/v1 @@ -649,79 +753,78 @@ spec: Apply the resource on `demo-controller`: ```bash -kubectl apply -f postgres.yaml --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl apply -f postgres.yaml --context demo-controller --kubeconfig $HOME/.kube/config ``` ### Step 7: Verify the Deployment -1. **Check Postgres Resource and Pods on `demo-controller`**: +#### 1. Check Postgres Resource and Pods on `demo-controller` ```bash -kubectl get pg,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config +$ kubectl get pg,pods,secret -n demo --context demo-controller --kubeconfig $HOME/.kube/config ``` -**Output**: - ```shell - NAME VERSION STATUS AGE - postgres.kubedb.com/postgres 11.5.2 Ready 99s - - NAME READY STATUS RESTARTS AGE - pod/postgres-0 3/3 Running 0 95s - pod/postgres-2 3/3 Running 0 95s - - NAME TYPE DATA AGE - secret/postgres-auth kubernetes.io/basic-auth 2 95s - - ``` +**Output:** +```bash +NAME VERSION STATUS AGE +postgres.kubedb.com/postgres 17.2 Ready 99s -2. **Check Pods and Secrets on `demo-worker`**: +NAME READY STATUS RESTARTS AGE +pod/postgres-0 2/2 Running 0 95s +pod/postgres-2 2/2 Running 0 95s -```bash -kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config +NAME TYPE DATA AGE +secret/postgres-auth kubernetes.io/basic-auth 2 95s ``` -**Output**: +#### 2. Check Pods and Secrets on `demo-worker` + +```bash +$ kubectl get pods,secrets -n demo --context demo-worker --kubeconfig $HOME/.kube/config ``` -NAME READY STATUS RESTARTS AGE -postgres-1 3/3 Running 0 95s -NAME TYPE DATA AGE -secret/postgres-auth kubernetes.io/basic-auth 2 95s +**Output:** + +```bash +NAME READY STATUS RESTARTS AGE +pod/postgres-1 2/2 Running 0 95s + +NAME TYPE DATA AGE +secret/postgres-auth kubernetes.io/basic-auth 2 95s ``` -3. **Verify Cluster Status**: - Connect to a Postgres pod and check cluster status using SQL queries or KubeDB status fields. +#### 3. Verify Replication Status -```shell -➤ kubectl exec -it -n demo ha-postgres-1 -- bash +Connect to the primary Postgres pod and check the replication status: + +```bash +$ kubectl exec -it -n demo pod/postgres-0 --context demo-controller -- bash Defaulted container "postgres" out of: postgres, pg-coordinator, postgres-init-container (init) -ha-postgres-1:/$ psql -psql (16.8) -Type "help" for help. +postgres-0:/$ psql -U postgres +``` + +Run the following query: -postgres=# select * from pg_stat_replication; +```sql +SELECT * FROM pg_stat_replication; +``` + +**Output:** + +``` pid | usesysid | usename | application_name | client_addr | client_hostname | client_port | backend_start | backend_xmin | state | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state | reply_time ------+----------+----------+------------------+-------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------------+----------------+-----------------+---------------+------------+------------------------------- - 495 | 10 | postgres | ha-postgres-0 | 10.42.0.78 | | 49492 | 2025-08-26 12:03:47.940257+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000212 | 00:00:00.00207 | 00:00:00.00212 | 0 | async | 2025-08-26 12:05:33.011941+00 - 1183 | 10 | postgres | ha-postgres-2 | 10.42.0.82 | | 58254 | 2025-08-26 12:05:28.257383+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000179 | 00:00:00.00185 | 00:00:00.001912 | 0 | async | 2025-08-26 12:05:33.01175+00 + 495 | 10 | postgres | postgres-1 | 10.1.0.12 | | 49492 | 2025-08-26 12:03:47.940257+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000212 | 00:00:00.00207 | 00:00:00.00212 | 0 | async | 2025-08-26 12:05:33.011941+00 + 1183 | 10 | postgres | postgres-2 | 10.42.0.82 | | 58254 | 2025-08-26 12:05:28.257383+00 | | streaming | 0/50001B0 | 0/50001B0 | 0/50001B0 | 0/50001B0 | 00:00:00.000179 | 00:00:00.00185 | 00:00:00.001912 | 0 | async | 2025-08-26 12:05:33.01175+00 (2 rows) - - ``` -## Troubleshooting Tips - -- **Pods Not Running**: Check pod logs for errors related to storage, networking, or configuration. -- **Cluster Not Joining**: Ensure the `RawFeedbackJsonString` feature gate is enabled and verify network connectivity between clusters. -- **KubeSlice Issues**: Confirm that the network interface matches your cluster's configuration and that sidecar containers are injected. -- **Postgres Not Synced**: Check replication status and ensure all nodes are part of the cluster. +Both `postgres-1` (on `demo-worker`, connected via KubeSlice) and `postgres-2` (on `demo-controller`) should appear in streaming replication state. ## Next Steps -- **Accessing the Database**: Use the generated secret to retrieve credentials and connect to the Postgres instance. +- **Accessing the Database**: Use the `postgres-auth` secret to retrieve credentials and connect to the Postgres instance. - **Scaling**: Adjust the `PlacementPolicy` to add or remove replicas across clusters. - **Monitoring**: Integrate KubeDB with monitoring tools like Prometheus for cluster health insights. -For further details, refer to the [KubeDB Documentation](https://kubedb.com/docs/v2025.7.31/) -