Skip to content

fix(valkey): harden failover safety, node/sentinel identity, and secret handling #664

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mberlofa merged 5 commits into from
Jul 4, 2026
Merged

fix(valkey): harden failover safety, node/sentinel identity, and secret handling #664

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dba2664
fix(valkey): harden failover safety, node/sentinel identity, and secr…
mreho Jul 3, 2026
f0763b2
test(valkey): updated tests for sentinel failover hardening
mreho Jul 3, 2026
fcda7d0
fix(valkey): added publishNotReadyAddresses for sentinel master boots…
mreho Jul 4, 2026
e78717b
fix(valkey): keep the Sentinel wait below the startup budget
mreho Jul 4, 2026
c62670c
fix(valkey): revert sentinel publish-not-ready bootstrap change
mreho Jul 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions charts/valkey/docs/sentinel.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,15 @@ sentinel:
Anti-split-brain safety does not depend on the bootstrap marker surviving reschedules.
When Sentinel is unreachable, each node probes peer `INFO replication` before choosing a role.

Data loss is possible only if every data node restarts at the same time with no peers online.
Enable `node.persistence.enabled=true` when you need RDB/AOF to survive pod reschedules.
Without additional safeguards, a single restart of the active master pod inside the
`down-after-milliseconds` window would resume it as an empty master and make every
replica full-resync from an empty dataset. The chart mitigates this with
`sentinel.gracefulFailover` (preStop-triggered failover on voluntary disruptions) and
`sentinel.startupFailoverGuard` (a fresh master refuses to resume while peers still
hold data and forces a failover first). Residual data loss remains possible if these
guards are disabled, if no replica is promotable, or if every data node restarts at
the same time. Enable `node.persistence.enabled=true` when you need RDB/AOF to
survive pod reschedules.

### Fail-closed with persistence enabled

Expand Down
16 changes: 16 additions & 0 deletions charts/valkey/templates/_helpers.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,3 +484,19 @@ topologySpreadConstraints:
{{- end }}
{{- end }}
{{- end -}}

{{/*
Probe commands for sentinel-mode pods. Auth flows through the VALKEYCLI_AUTH
environment variable set on the container, never through argv.
*/}}
{{- define "valkey.nodeProbeCommand" -}}
valkey-cli {{ include "valkey.probeTlsArgs" . }} -p {{ .Values.service.ports.valkey }} ping
{{- end -}}

{{- define "valkey.sentinelProbeCommand" -}}
valkey-cli {{ include "valkey.probeTlsArgs" . }} -p {{ .Values.service.ports.sentinel }} ping
{{- end -}}

{{- define "valkey.sentinelReadyCommand" -}}
valkey-cli {{ include "valkey.probeTlsArgs" . }} -p {{ .Values.service.ports.sentinel }} sentinel get-master-addr-by-name {{ .Values.sentinel.masterSet }} | grep -q .
{{- end -}}
Loading