Skip to content

fix: correlate member status with health report by member ID#392

Open
xrl wants to merge 1 commit into
etcd-io:mainfrom
xrl:pr/fix-leader-status
Open

fix: correlate member status with health report by member ID#392
xrl wants to merge 1 commit into
etcd-io:mainfrom
xrl:pr/fix-leader-status

Conversation

@xrl

@xrl xrl commented Jul 2, 2026

Copy link
Copy Markdown

On a healthy 3-member cluster, status.leaderID reported one member (799d0cd98f5cac87) while status.members[] simultaneously marked a different member with isLeader: true. updateStatus paired memberListResp.Members (ordered by member ID) with memberHealth (ordered by endpoint) by index, so isLeader/isHealthy/version could attach to the wrong member. Health entries are now looked up by Status.Header.MemberId, and both members[].isLeader and status.leaderID derive from the same FindLeaderStatus result. Added a unit test with an ID-ordered member list and endpoint-ordered health report that fails on the previous code and passes with this change.


PR series — operability fixes & TLS

Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.

PR Lands After
Reviewable now — small, any order
🟢 #374 Requeue instead of swallowing the client-cert provisioning error
🟢 #391 Grant events.k8s.io RBAC so operator Events are actually recorded
🟢→ #392 Correlate members[]/leaderID from one health snapshot (consistent leader)
🟢 #393 Propagate user-supplied altNames.ipAddresses into certificates
🟢 #394 Accept day-suffix validityDuration (365d, 100d12h) as documented
🟢 #395 Surface early reconcile errors as a Degraded condition (was empty status)
🟢 #379 Configurable reconcile worker pool (--max-concurrent-reconciles)
🟢 #369 kind-based stress/scale e2e harness (1/3/7 members, churn, quorum watcher)
TLS stack — in order
🟢 #376 Independent spec.tls.{peer,client} surfaces (breaking alpha API)
#377 TLSReady condition + TLS lifecycle Events #376
#378 Multi-member TLS quorum e2e + PeerCANotShared #377
Parked as drafts pending #363
#382 Per-cluster domain metrics on the operator /metrics endpoint
#384 EtcdCluster admission webhooks (consolidating with #328) #363
#386 EtcdBackup CR → object storage (S3/GCS) #363
#387 Automatic quorum-loss disaster recovery (bootstrap-latch guarded) #363

🟢 ready · ⚪ draft

The member list returned by etcd is ordered by member ID while the
health report is ordered by endpoint, so pairing the two by index can
attach IsLeader, IsHealthy and Version to the wrong member. This made
members[].isLeader disagree with status.leaderID on healthy clusters.

Look up health info by Status.Header.MemberId and derive both
members[].isLeader and status.leaderID from the same FindLeaderStatus
result.

Signed-off-by: Xavier Lange <xrlange@gmail.com>
@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: xrl
Once this PR has been reviewed and has the lgtm label, please assign ivanvc for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow

Copy link
Copy Markdown

Hi @xrl. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant