Skip to content

Replication broken on Kubernetes < 1.28 #160

Description

@mephs

Summary

On Kubernetes clusters older than 1.28, enabling replication (replica.enabled=true) results in all pods running as independent masters instead of a primary-replica topology.

Root cause

The startup script determines pod role from the POD_INDEX environment variable, which is populated via the downward API from the apps.kubernetes.io/pod-index pod label:

POD_INDEX=${POD_INDEX:-0}
IS_MASTER=false

if [ "$POD_INDEX" = "0" ]; then
  IS_MASTER=true
  ...

The apps.kubernetes.io/pod-index label is only set automatically by the StatefulSet controller starting from Kubernetes 1.28 by default. On older clusters the label is absent, the env var resolves to the default 0.

Reproduction

  1. Deploy the chart with replication on any Kubernetes < 1.28 cluster:
   helm install valkey valkey/valkey \
     --set replica.enabled=true \
     --set replica.persistence.size=1Gi
  1. Wait for all pods to become ready.
  2. Check replication role on each pod:
   for i in {0..2}; do
     echo "=== pod $i ==="
     kubectl exec valkey-$i -- valkey-cli INFO replication | grep role
   done

Expected

  • pod-0: role:master
  • pod-1, pod-2: role:slave, connected to pod-0

Actual

All three pods report role:master with connected_slaves:0. Writes to the write-service land on pod-0 only; reads from the read-service round-robin across three unrelated datasets.

Suggested fix

Fall back to parsing the pod ordinal from $HOSTNAME when POD_INDEX is not set by the downward API.

POD_INDEX=${POD_INDEX:-${HOSTNAME##*-}}

This keeps the existing behavior on 1.28+ (where POD_INDEX is populated from the label) and restores correct behavior on older clusters.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions