Summary
On Kubernetes clusters older than 1.28, enabling replication (replica.enabled=true) results in all pods running as independent masters instead of a primary-replica topology.
Root cause
The startup script determines pod role from the POD_INDEX environment variable, which is populated via the downward API from the apps.kubernetes.io/pod-index pod label:
POD_INDEX=${POD_INDEX:-0}
IS_MASTER=false
if [ "$POD_INDEX" = "0" ]; then
IS_MASTER=true
...
The apps.kubernetes.io/pod-index label is only set automatically by the StatefulSet controller starting from Kubernetes 1.28 by default. On older clusters the label is absent, the env var resolves to the default 0.
Reproduction
- Deploy the chart with replication on any Kubernetes < 1.28 cluster:
helm install valkey valkey/valkey \
--set replica.enabled=true \
--set replica.persistence.size=1Gi
- Wait for all pods to become ready.
- Check replication role on each pod:
for i in {0..2}; do
echo "=== pod $i ==="
kubectl exec valkey-$i -- valkey-cli INFO replication | grep role
done
Expected
- pod-0:
role:master
- pod-1, pod-2:
role:slave, connected to pod-0
Actual
All three pods report role:master with connected_slaves:0. Writes to the write-service land on pod-0 only; reads from the read-service round-robin across three unrelated datasets.
Suggested fix
Fall back to parsing the pod ordinal from $HOSTNAME when POD_INDEX is not set by the downward API.
POD_INDEX=${POD_INDEX:-${HOSTNAME##*-}}
This keeps the existing behavior on 1.28+ (where POD_INDEX is populated from the label) and restores correct behavior on older clusters.
Summary
On Kubernetes clusters older than 1.28, enabling replication (
replica.enabled=true) results in all pods running as independent masters instead of a primary-replica topology.Root cause
The startup script determines pod role from the
POD_INDEXenvironment variable, which is populated via the downward API from theapps.kubernetes.io/pod-indexpod label:The
apps.kubernetes.io/pod-indexlabel is only set automatically by the StatefulSet controller starting from Kubernetes 1.28 by default. On older clusters the label is absent, the env var resolves to the default0.Reproduction
helm install valkey valkey/valkey \ --set replica.enabled=true \ --set replica.persistence.size=1GiExpected
role:masterrole:slave, connected to pod-0Actual
All three pods report
role:masterwithconnected_slaves:0. Writes to the write-service land on pod-0 only; reads from the read-service round-robin across three unrelated datasets.Suggested fix
Fall back to parsing the pod ordinal from
$HOSTNAMEwhenPOD_INDEXis not set by the downward API.POD_INDEX=${POD_INDEX:-${HOSTNAME##*-}}This keeps the existing behavior on 1.28+ (where
POD_INDEXis populated from the label) and restores correct behavior on older clusters.