fix: correlate member status with health report by member ID#392
Conversation
The member list returned by etcd is ordered by member ID while the health report is ordered by endpoint, so pairing the two by index can attach IsLeader, IsHealthy and Version to the wrong member. This made members[].isLeader disagree with status.leaderID on healthy clusters. Look up health info by Status.Header.MemberId and derive both members[].isLeader and status.leaderID from the same FindLeaderStatus result. Signed-off-by: Xavier Lange <xrlange@gmail.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xrl The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi @xrl. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
On a healthy 3-member cluster,
status.leaderIDreported one member (799d0cd98f5cac87) whilestatus.members[]simultaneously marked a different member withisLeader: true.updateStatuspairedmemberListResp.Members(ordered by member ID) withmemberHealth(ordered by endpoint) by index, soisLeader/isHealthy/versioncould attach to the wrong member. Health entries are now looked up byStatus.Header.MemberId, and bothmembers[].isLeaderandstatus.leaderIDderive from the sameFindLeaderStatusresult. Added a unit test with an ID-ordered member list and endpoint-ordered health report that fails on the previous code and passes with this change.PR series — operability fixes & TLS
Small single-purpose PRs from live kind-cluster testing of the operator. Each stands alone unless an After is listed. → = this PR.
events.k8s.ioRBAC so operator Events are actually recordedmembers[]/leaderIDfrom one health snapshot (consistent leader)altNames.ipAddressesinto certificatesvalidityDuration(365d,100d12h) as documentedDegradedcondition (was empty status)--max-concurrent-reconciles)spec.tls.{peer,client}surfaces (breaking alpha API)TLSReadycondition + TLS lifecycle EventsPeerCANotShared/metricsendpointEtcdBackupCR → object storage (S3/GCS)🟢 ready · ⚪ draft