Skip to content

OCPBUGS-92013: fix NodePool stuck in UpdatingVersion/UpdatingConfig due to stale conversion-data annotation#8821

Merged
openshift-merge-bot[bot] merged 2 commits into
openshift:mainfrom
csrwng:fix/ocpbugs-92013-stale-conversion-data
Jun 25, 2026
Merged

OCPBUGS-92013: fix NodePool stuck in UpdatingVersion/UpdatingConfig due to stale conversion-data annotation#8821
openshift-merge-bot[bot] merged 2 commits into
openshift:mainfrom
csrwng:fix/ocpbugs-92013-stale-conversion-data

Conversation

@csrwng

@csrwng csrwng commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Replaces the cluster.x-k8s.io/conversion-data annotation-based v1beta2 cross-check in MachineDeploymentComplete with a direct read of Status.V1Beta2 fields
  • Fixes a permanent deadlock where NodePools get stuck reporting UpdatingVersion=True and UpdatingConfig=True after a rollout completes

Root cause

The conversion-data annotation is metadata. When CAPI updates MachineDeployment status via the v1beta2 status subresource, the API server's PrepareForUpdate preserves metadata from etcd — so the annotation is never refreshed by status writes. If the last main-resource v1beta2 write captured mid-rollout state (e.g. replicas: 4, upToDateReplicas: null), the annotation becomes permanently stale. machineDeploymentCompleteFromConversionData then returns false forever, preventing reconcileMachineDeploymentStatus from updating nodePool.Status.Version and the config annotation.

Fix

The v1beta1 MachineDeploymentStatus already has a V1Beta2 nested field that stores the v1beta2-native replica counters (UpToDateReplicas, AvailableReplicas). These fields are part of the status (not metadata), so they are correctly updated on every status-subresource write via Convert_v1beta2_MachineDeploymentStatus_To_v1beta1_MachineDeploymentStatus.

The new machineDeploymentCompleteFromV1Beta2Status reads these fields directly instead of parsing the stale annotation.

Jira: https://issues.redhat.com/browse/OCPBUGS-92013

Test plan

  • Unit tests updated (TestMachineDeploymentComplete — 7 cases covering v1beta2 disagreement, nil V1Beta2, nil UpToDateReplicas, v1beta1-only fallback)
  • Full nodepool test suite passes (go test ./hypershift-operator/controllers/nodepool/)
  • e2e-aws-upgrade-hypershift-operator (CI)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Improved MachineDeployment completion detection to rely on native status replica fields instead of conversion-data annotation parsing, resulting in more reliable readiness checks.
    • Preserved backward-compatible behavior when newer status details are unavailable.
    • Enhanced handling of replica edge cases, including scenarios where replica fields are nil.
  • Tests
    • Updated and expanded unit tests to validate completion logic using the v1beta2 status replica fields, including explicit nil pointer cases.

…on for MachineDeploymentComplete

The conversion-data annotation is metadata, which is preserved from etcd
on status-subresource writes. Since CAPI updates MachineDeployment
status via the status subresource, the annotation is never refreshed by
those writes. If the last main-resource v1beta2 write captured mid-rollout
state, the annotation becomes permanently stale, causing
MachineDeploymentComplete to return false forever and deadlocking the
NodePool in UpdatingVersion/UpdatingConfig.

Replace the annotation-based cross-check with a read of
Status.V1Beta2.UpToDateReplicas and Status.V1Beta2.AvailableReplicas.
These fields are part of the v1beta1 Status struct (not metadata), so
they are correctly updated on every status-subresource write via the
v1beta2-to-v1beta1 conversion layer.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@csrwng: This pull request references Jira Issue OCPBUGS-92013, which is invalid:

  • expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

  • Replaces the cluster.x-k8s.io/conversion-data annotation-based v1beta2 cross-check in MachineDeploymentComplete with a direct read of Status.V1Beta2 fields
  • Fixes a permanent deadlock where NodePools get stuck reporting UpdatingVersion=True and UpdatingConfig=True after a rollout completes

Root cause

The conversion-data annotation is metadata. When CAPI updates MachineDeployment status via the v1beta2 status subresource, the API server's PrepareForUpdate preserves metadata from etcd — so the annotation is never refreshed by status writes. If the last main-resource v1beta2 write captured mid-rollout state (e.g. replicas: 4, upToDateReplicas: null), the annotation becomes permanently stale. machineDeploymentCompleteFromConversionData then returns false forever, preventing reconcileMachineDeploymentStatus from updating nodePool.Status.Version and the config annotation.

Fix

The v1beta1 MachineDeploymentStatus already has a V1Beta2 nested field that stores the v1beta2-native replica counters (UpToDateReplicas, AvailableReplicas). These fields are part of the status (not metadata), so they are correctly updated on every status-subresource write via Convert_v1beta2_MachineDeploymentStatus_To_v1beta1_MachineDeploymentStatus.

The new machineDeploymentCompleteFromV1Beta2Status reads these fields directly instead of parsing the stale annotation.

Jira: https://issues.redhat.com/browse/OCPBUGS-92013

Test plan

  • Unit tests updated (TestMachineDeploymentComplete — 7 cases covering v1beta2 disagreement, nil V1Beta2, nil UpToDateReplicas, v1beta1-only fallback)
  • Full nodepool test suite passes (go test ./hypershift-operator/controllers/nodepool/)
  • e2e-aws-upgrade-hypershift-operator (CI)

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@csrwng

csrwng commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jun 24, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@csrwng: This pull request references Jira Issue OCPBUGS-92013, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot requested review from Nirshal and bryan-cox June 24, 2026 14:03
@openshift-ci openshift-ci Bot added area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed do-not-merge/needs-area labels Jun 24, 2026
@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: bdd5ab72-b890-4ea4-8e1d-c50ca8e965d0

📥 Commits

Reviewing files that changed from the base of the PR and between 5ae09a3 and d2cf0ff.

📒 Files selected for processing (2)
  • hypershift-operator/controllers/nodepool/capi_test.go
  • hypershift-operator/controllers/nodepool/nodepool_controller.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • hypershift-operator/controllers/nodepool/capi_test.go
  • hypershift-operator/controllers/nodepool/nodepool_controller.go

📝 Walkthrough

Walkthrough

MachineDeploymentComplete in nodepool_controller.go now checks MachineDeploymentStatus.V1Beta2 instead of parsing the conversion-data annotation JSON. It returns true when Status.V1Beta2 is nil, and otherwise compares UpToDateReplicas and AvailableReplicas with the desired replica count. The related JSON and conversion imports were removed. TestMachineDeploymentComplete was updated to use V1Beta2 fields directly and to cover nil status and nil replica field cases.

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main fix: NodePool no longer depends on stale conversion-data annotations for completion checks.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All test titles in the changed test file are static literals; none include generated, time-based, or other dynamic values.
Test Structure And Quality ✅ Passed TestMachineDeploymentComplete is table-driven unit code; each subtest checks one outcome, with no cluster setup/cleanup or Eventually/Consistently waits.
Topology-Aware Scheduling Compatibility ✅ Passed Only MachineDeployment completion/status logic and tests changed; no pod scheduling constraints, selectors, affinities, spreads, or PDBs were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed PASS: The change only updates unit tests and an internal completion helper; no Ginkgo e2e tests, IPv4-only assumptions, or external connectivity calls were added.
No-Weak-Crypto ✅ Passed No crypto primitives or secret comparisons were added; the patch only changes MachineDeployment completion logic and tests.
Container-Privileges ✅ Passed PR only changes Go controller/test files; no Kubernetes manifests or privilege-related fields were added.
No-Sensitive-Data-In-Logs ✅ Passed The PR only changes completion logic and unit tests; no new logging or sensitive values were added, and the updated helper contains no log calls.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci-robot

Copy link
Copy Markdown

@csrwng: This pull request references Jira Issue OCPBUGS-92013, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

Summary

  • Replaces the cluster.x-k8s.io/conversion-data annotation-based v1beta2 cross-check in MachineDeploymentComplete with a direct read of Status.V1Beta2 fields
  • Fixes a permanent deadlock where NodePools get stuck reporting UpdatingVersion=True and UpdatingConfig=True after a rollout completes

Root cause

The conversion-data annotation is metadata. When CAPI updates MachineDeployment status via the v1beta2 status subresource, the API server's PrepareForUpdate preserves metadata from etcd — so the annotation is never refreshed by status writes. If the last main-resource v1beta2 write captured mid-rollout state (e.g. replicas: 4, upToDateReplicas: null), the annotation becomes permanently stale. machineDeploymentCompleteFromConversionData then returns false forever, preventing reconcileMachineDeploymentStatus from updating nodePool.Status.Version and the config annotation.

Fix

The v1beta1 MachineDeploymentStatus already has a V1Beta2 nested field that stores the v1beta2-native replica counters (UpToDateReplicas, AvailableReplicas). These fields are part of the status (not metadata), so they are correctly updated on every status-subresource write via Convert_v1beta2_MachineDeploymentStatus_To_v1beta1_MachineDeploymentStatus.

The new machineDeploymentCompleteFromV1Beta2Status reads these fields directly instead of parsing the stale annotation.

Jira: https://issues.redhat.com/browse/OCPBUGS-92013

Test plan

  • Unit tests updated (TestMachineDeploymentComplete — 7 cases covering v1beta2 disagreement, nil V1Beta2, nil UpToDateReplicas, v1beta1-only fallback)
  • Full nodepool test suite passes (go test ./hypershift-operator/controllers/nodepool/)
  • e2e-aws-upgrade-hypershift-operator (CI)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
  • Improved MachineDeployment completion checks to use native status fields, making readiness detection more reliable.
  • Added support for older environments by treating missing status details as complete when appropriate.
  • Expanded validation coverage for replica count edge cases, including nil values.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@hypershift-operator/controllers/nodepool/nodepool_controller.go`:
- Around line 816-818: The completion check in NodePool reconciliation is
incorrectly treating nil v1beta2 replica status fields as zero when comparing
against deployment.Spec.Replicas, which can incorrectly mark a scale-to-zero
MachineDeployment as complete. Update the logic in the NodePool controller’s
replica comparison path to require UpToDateReplicas and AvailableReplicas to be
present before comparing values, rather than using ptr.Deref defaults, and keep
the desired replica comparison in sync with that presence check. Add a test for
the zero-replica case where v1beta2 status fields are nil to verify it is not
considered complete.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 479d46e9-9a2a-4496-beb2-c77137789c23

📥 Commits

Reviewing files that changed from the base of the PR and between bc3bda9 and 5ae09a3.

📒 Files selected for processing (2)
  • hypershift-operator/controllers/nodepool/capi_test.go
  • hypershift-operator/controllers/nodepool/nodepool_controller.go

Comment thread hypershift-operator/controllers/nodepool/nodepool_controller.go Outdated
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 42.55%. Comparing base (bc3bda9) to head (d2cf0ff).
⚠️ Report is 30 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8821      +/-   ##
==========================================
- Coverage   42.55%   42.55%   -0.01%     
==========================================
  Files         768      768              
  Lines       95297    95283      -14     
==========================================
- Hits        40558    40544      -14     
  Misses      51932    51932              
  Partials     2807     2807              
Files with missing lines Coverage Δ
...erator/controllers/nodepool/nodepool_controller.go 42.26% <100.00%> (-0.92%) ⬇️
Flag Coverage Δ
cmd-support 35.46% <ø> (ø)
cpo-hostedcontrolplane 44.84% <ø> (ø)
cpo-other 44.70% <ø> (ø)
hypershift-operator 53.02% <100.00%> (-0.03%) ⬇️
other 31.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…g to zero

Nil UpToDateReplicas/AvailableReplicas means "not yet set", not "zero
replicas". Using ptr.Deref with a zero default would incorrectly match
a desired count of zero, marking incomplete deployments as complete.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jun 24, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

@openshift-ci

openshift-ci Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox, csrwng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@csrwng

csrwng commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

/hold

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 24, 2026
@hypershift-jira-solve-ci

Copy link
Copy Markdown

Now I have a very clear picture. The key observations:

  1. Line 597: "No machines associated with infra id node-pool-dsphk were found" — For TestNodePool/HostedCluster2, no machines were ever created. The NodePool has replicas: 0 in its spec, meaning nodes were never requested.

  2. Line 575-576: For TestNodePoolAutoscalingScaleFromZero, the conditions show NoWorkerNodesAvailable — this test starts with replicas=0 and is supposed to scale up from zero via autoscaler, but it never gets past the ValidateHostedCluster phase because DNS never resolves.

  3. Line 644: For TestKarpenterUpgradeControlPlane, similarly NoWorkerNodesAvailable.

The root cause is clear: ExternalDNS failed to create DNS records for these 3 hosted clusters within the 10-minute timeout. The ExternalDNS operator didn't create Route53 records for api-scale-from-zero-dxq96, api-node-pool-dsphk, and api-karpenter-upgrade-control-plane-ntjdj subdomain records, while it did (eventually) create records for all other hosted clusters (~10 others that all passed).

This is a transient ExternalDNS/Route53 propagation failure — a flaky CI infrastructure issue, not caused by the PR changes (which only affect NodePool controller's MachineDeploymentComplete logic and conversion-data annotation handling).

Now I have enough information to write the final report.

Test Failure Analysis Complete

Job Information

  • Prow Job: pull-ci-openshift-hypershift-main-e2e-aws-4-22
  • Build ID: 2069786993973268480
  • Target: e2e-aws-4-22
  • Result: 529 tests run, 29 skipped, 7 failures (3 unique test failures + their parent wrappers)

Test Failure Analysis

Error

dial tcp: lookup api-{cluster-name}.service.ci.hypershift.devcluster.openshift.com on 172.30.0.10:53: no such host
util.go:384: Failed to wait for a successful connection to the guest API server in 10m0s: context deadline exceeded
ExternalDNSReachable=False: ExternalDNSHostNotReachable

Summary

Three hosted clusters (scale-from-zero-dxq96, node-pool-dsphk, karpenter-upgrade-control-plane-ntjdj) failed because ExternalDNS never created their Route53 DNS records within the 10-minute ValidateHostedCluster timeout. The control planes were fully available (KubeAPIServer, etcd, infrastructure all healthy), but the tests could not reach them because DNS names never resolved. All ~10 other hosted clusters in the same run experienced the same initial DNS resolution failure but recovered within 2-3 minutes. This is a transient ExternalDNS/Route53 propagation flake unrelated to the PR #8821 changes, which only modify nodepool_controller.go logic for MachineDeploymentComplete and the conversion-data annotation.

Root Cause

Transient ExternalDNS/Route53 propagation failure — a CI infrastructure flake.

All hosted clusters in this job initially failed DNS resolution (no such host). The passing tests progressed through three phases: (1) no such host → (2) i/o timeout at resolved IP (DNS propagated, but NLB endpoint not ready) → (3) successful connection, typically within 2-3 minutes. The three failing clusters never progressed past phase 1 — their DNS records were never created or propagated by ExternalDNS within the 10-minute window.

Evidence that the control planes themselves were healthy for the failing clusters:

  • KubeAPIServerAvailable: True, EtcdAvailable: True, InfrastructureReady: True, Available: True
  • AWSEndpointAvailable: True, AWSEndpointServiceAvailable: True
  • The only failing conditions were ExternalDNSReachable: False and downstream effects (NoWorkerNodesAvailable, ClusterOperatorsNotAvailable)

The PR #8821 changes are limited to nodepool_controller.go (replacing conversion-data annotation check with direct Status.V1Beta2 field reads in MachineDeploymentComplete) and its unit test. These changes do not affect ExternalDNS, HostedCluster creation, DNS record management, or the ValidateHostedCluster test framework — the failure mechanism is entirely orthogonal to the code changes.

Recommendations
  1. Retry the job — this is a transient ExternalDNS/Route53 flake. The PR changes are unrelated to the failure.
  2. No code changes needed — the PR (OCPBUGS-92013: fix NodePool stuck in UpdatingVersion/UpdatingConfig due to stale conversion-data annotation #8821) modifies only NodePool controller logic for MachineDeploymentComplete and does not touch ExternalDNS, DNS record creation, or hosted cluster lifecycle.
  3. For persistent flakiness — if this pattern recurs across multiple runs, investigate the shared ExternalDNS deployment on the management cluster (ci.hypershift.devcluster.openshift.com) for rate limiting, throttling, or resource exhaustion when 10+ hosted clusters are created simultaneously.
Evidence
Evidence Detail
Failing tests TestNodePoolAutoscalingScaleFromZero, TestKarpenterUpgradeControlPlane, TestNodePool/HostedCluster2 (+ parent wrappers = 7 total)
Failure mode ExternalDNSReachable=False — DNS records never created for 3 of ~13 hosted clusters
Passing tests DNS timing Connected in 2m6s–2m38s after initial no such host errors
Failing tests DNS timing Never resolved in 10m0s — only ever no such host, never reached i/o timeout phase
Control plane status All healthy: KubeAPIServerAvailable=True, EtcdAvailable=True, Available=True, AWSEndpointAvailable=True
NodePool node-pool-dsphk spec.replicas: 0, status.replicas: 0 — no machines created because test never got past ValidateHostedCluster
NodePool scale-from-zero-dxq96 spec.replicas: 1, status.replicas: 0 — machine created but node Ready=False; DNS still not reachable
PR #8821 scope Only modifies nodepool_controller.go and capi_test.go — no ExternalDNS, DNS, or HC lifecycle changes
Management cluster fcc8f11dc5-mgmt in clusters namespace — shared CI infrastructure
ExternalDNS domain service.ci.hypershift.devcluster.openshift.com

@cwbotbot

Copy link
Copy Markdown

Test Results

e2e-aws

e2e-aks

@csrwng

csrwng commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

The aws-4-22 failure was an infra issue

ExternalDNS failed to create DNS records for these 3 hosted clusters within the 10-minute timeout.

/retest-required
/hold cancel

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 25, 2026
@csrwng

csrwng commented Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

/verified by @csrwng
Ran a manual test to simulate a stuck nodepool with a stale annotation. Updating the HO to the version from this PR resulted in the UpdatingVersion and UpdatingConfig conditions on the nodepool to be set to false (as expected)

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 25, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@csrwng: This PR has been marked as verified by @csrwng.

Details

In response to this:

/verified by @csrwng
Ran a manual test to simulate a stuck nodepool with a stale annotation. Updating the HO to the version from this PR resulted in the UpdatingVersion and UpdatingConfig conditions on the nodepool to be set to false (as expected)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

@csrwng: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot Bot merged commit 6de31aa into openshift:main Jun 25, 2026
41 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@csrwng: Jira Issue OCPBUGS-92013: Some pull requests linked via external trackers have merged:

The following pull request, linked via external tracker, has not merged:

All associated pull requests must be merged or unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-92013 has not been moved to the MODIFIED state.

This PR is marked as verified. If the remaining PRs listed above are marked as verified before merging, the issue will automatically be moved to VERIFIED after all of the changes from the PRs are available in an accepted nightly payload.

Details

In response to this:

Summary

  • Replaces the cluster.x-k8s.io/conversion-data annotation-based v1beta2 cross-check in MachineDeploymentComplete with a direct read of Status.V1Beta2 fields
  • Fixes a permanent deadlock where NodePools get stuck reporting UpdatingVersion=True and UpdatingConfig=True after a rollout completes

Root cause

The conversion-data annotation is metadata. When CAPI updates MachineDeployment status via the v1beta2 status subresource, the API server's PrepareForUpdate preserves metadata from etcd — so the annotation is never refreshed by status writes. If the last main-resource v1beta2 write captured mid-rollout state (e.g. replicas: 4, upToDateReplicas: null), the annotation becomes permanently stale. machineDeploymentCompleteFromConversionData then returns false forever, preventing reconcileMachineDeploymentStatus from updating nodePool.Status.Version and the config annotation.

Fix

The v1beta1 MachineDeploymentStatus already has a V1Beta2 nested field that stores the v1beta2-native replica counters (UpToDateReplicas, AvailableReplicas). These fields are part of the status (not metadata), so they are correctly updated on every status-subresource write via Convert_v1beta2_MachineDeploymentStatus_To_v1beta1_MachineDeploymentStatus.

The new machineDeploymentCompleteFromV1Beta2Status reads these fields directly instead of parsing the stale annotation.

Jira: https://issues.redhat.com/browse/OCPBUGS-92013

Test plan

  • Unit tests updated (TestMachineDeploymentComplete — 7 cases covering v1beta2 disagreement, nil V1Beta2, nil UpToDateReplicas, v1beta1-only fallback)
  • Full nodepool test suite passes (go test ./hypershift-operator/controllers/nodepool/)
  • e2e-aws-upgrade-hypershift-operator (CI)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
  • Improved MachineDeployment completion detection to rely on native status replica fields instead of conversion-data annotation parsing, resulting in more reliable readiness checks.
  • Preserved backward-compatible behavior when newer status details are unavailable.
  • Enhanced handling of replica edge cases, including scenarios where replica fields are nil.
  • Tests
  • Updated and expanded unit tests to validate completion logic using the v1beta2 status replica fields, including explicit nil pointer cases.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants