CNTRLPLANE-3661: add etcd data re-encryption after encryption key rotation (#8219) by muraee · Pull Request #8790 · openshift/hypershift

muraee · 2026-06-22T10:22:14Z

manual backport of #8219

openshift-merge-bot · 2026-06-22T10:22:17Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

coderabbitai · 2026-06-22T10:22:23Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 3629c987-680e-477a-a700-27033c59c11d

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

codecov · 2026-06-22T10:41:24Z

Codecov Report

❌ Patch coverage is 54.74453% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 35.71%. Comparing base (5cb8735) to head (fafc784).
⚠️ Report is 11 commits behind head on release-4.22.

Files with missing lines	Patch %	Lines
...rator/controllers/hostedcontrolplane/v2/kas/kms.go	56.04%	28 Missing and 12 partials ⚠️
...ontrollers/hostedcontrolplane/v2/kas/deployment.go	0.00%	15 Missing ⚠️
...or/controllers/hostedcontrolplane/v2/kas/aescbc.go	66.66%	2 Missing and 2 partials ⚠️
...r/controllers/hostedcontrolplane/v2/kas/kms/aws.go	81.25%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@               Coverage Diff                @@
##           release-4.22    #8790      +/-   ##
================================================
+ Coverage         35.45%   35.71%   +0.26%     
================================================
  Files               767      774       +7     
  Lines             93724    94614     +890     
================================================
+ Hits              33226    33795     +569     
- Misses            57785    58050     +265     
- Partials           2713     2769      +56

Files with missing lines	Coverage Δ
...ostedcontrolplane/hostedcontrolplane_controller.go	`36.78% <100.00%> (+0.02%)`	⬆️
...ontrollers/hostedcontrolplane/v2/cvo/deployment.go	`41.17% <100.00%> (+0.58%)`	⬆️
...controllers/hostedcontrolplane/v2/kas/kms/azure.go	`6.85% <ø> (-0.33%)`	⬇️
...trollers/hostedcontrolplane/v2/kas/kms/ibmcloud.go	`5.47% <ø> (-0.03%)`	⬇️
...r/controllers/hostedcontrolplane/v2/kas/kms/kms.go	`100.00% <ø> (ø)`
...lers/hostedcontrolplane/v2/kas/secretencryption.go	`53.60% <ø> (+25.69%)`	⬆️
...lane/v2/kube_storage_version_migrator/component.go	`0.00% <ø> (ø)`
...-plane-operator/hostedclusterconfigoperator/cmd.go	`0.00% <ø> (ø)`
...configoperator/controllers/reencryption/metrics.go	`88.46% <ø> (ø)`
...goperator/controllers/reencryption/reencryption.go	`69.03% <ø> (ø)`
... and 11 more

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

- Add a re-encryption controller in the HCCO that detects encryption key rotations and triggers StorageVersionMigration CRs to re-encrypt all existing etcd data with the new active key - Track progress through a new EtcdDataEncryptionUpToDate condition on HCP/HostedCluster - Vendor library-go's KubeStorageVersionMigrator and kube-storage-version-migrator informer/lister packages

openshift-ci-robot · 2026-06-22T13:07:38Z

@muraee: This pull request references CNTRLPLANE-3661 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.22.0" version, but no target version was set.

Details

In response to this:

manual backport of #8219

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

JoelSpeed · 2026-06-23T10:45:11Z

/approve for API

openshift-ci · 2026-06-23T10:45:47Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JoelSpeed, muraee

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [muraee]
~~api/OWNERS~~ [JoelSpeed]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ahitacat · 2026-06-25T14:01:28Z

/verified by ahitacat
Verified this in ARO-HCP personal dev environment.
Updated the HO with the latest build (with the this fix) , deployed a cluster with the build from this PR added as an annotation
quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22:on-pr-fafc784a49fad34c1759fca742860de75c369b09

Then I have rotate the key, and update the cluster. I could see the EtcdDataEncryptionUpToDate

    - lastTransitionTime: "2026-06-25T13:52:04Z"
      message: All etcd data is encrypted with the current active key
      observedGeneration: 2
      reason: ReEncryptionCompleted
      status: "True"
      type: EtcdDataEncryptionUpToDate

And the status.secretEncryption.activeKey is equal to the expected one:

status:
    secretEncryption:
      activeKey:
        azure:
          keyName: etcd-data-kms-encryption-key
          keyVaultName: ahitacat-kv-1fe591
          keyVersion: b6fd18e3c6d142d096aa587f09ff5879
        provider: Azure
      history:
      - completionTime: "2026-06-25T13:52:04Z"
        from:
          fingerprint: e79979acf6724eff9718433504dcefcc7c32b68b65c6a76599ba6683b72b536f
          provider: Azure
        startedTime: "2026-06-25T13:42:10Z"
        state: Completed
        to:
          fingerprint: 89d258d06a21c676d26c1d6474ca27de6f486ce8c467804f4ea68f1b788fdc5f
          provider: Azure

It seems OK enough to me to merge it.

openshift-ci-robot · 2026-06-25T14:01:42Z

@ahitacat: This PR has been marked as verified by ahitacat.

Details

In response to this:

/verified by ahitacat
Verified this in ARO-HCP personal dev environment.
Updated the HO with the latest build (with the this fix) , deployed a cluster with the build from this PR added as an annotation
quay.io/redhat-user-workloads/crt-redhat-acm-tenant/control-plane-operator-4-22:on-pr-fafc784a49fad34c1759fca742860de75c369b09

Then I have rotate the key, and update the cluster. I could see the EtcdDataEncryptionUpToDate
   - lastTransitionTime: "2026-06-25T13:52:04Z"
     message: All etcd data is encrypted with the current active key
     observedGeneration: 2
     reason: ReEncryptionCompleted
     status: "True"
     type: EtcdDataEncryptionUpToDate
And the status.secretEncryption.activeKey is equal to the expected one:
status:
   secretEncryption:
     activeKey:
       azure:
         keyName: etcd-data-kms-encryption-key
         keyVaultName: ahitacat-kv-1fe591
         keyVersion: b6fd18e3c6d142d096aa587f09ff5879
       provider: Azure
     history:
     - completionTime: "2026-06-25T13:52:04Z"
       from:
         fingerprint: e79979acf6724eff9718433504dcefcc7c32b68b65c6a76599ba6683b72b536f
         provider: Azure
       startedTime: "2026-06-25T13:42:10Z"
       state: Completed
       to:
         fingerprint: 89d258d06a21c676d26c1d6474ca27de6f486ce8c467804f4ea68f1b788fdc5f
         provider: Azure
It seems OK enough to me to merge it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-bot · 2026-06-25T15:21:42Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-21
/test e2e-aws-4-21
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

openshift-merge-bot · 2026-06-25T15:48:25Z

/retest-required

Remaining retests: 0 against base HEAD e3ad989 and 2 for PR HEAD fafc784 in total

openshift-merge-bot · 2026-06-26T00:46:30Z

/retest-required

Remaining retests: 0 against base HEAD f1fe6c1 and 1 for PR HEAD fafc784 in total

hypershift-jira-solve-ci · 2026-06-26T02:48:34Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-release-4.22-e2e-aws | Build: 2070308354055475200 | Cost: $5.302029050000001 | Failed step: hypershift-azure-run-e2e

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

openshift-merge-bot · 2026-06-26T08:44:53Z

/retest-required

Remaining retests: 0 against base HEAD 76d3035 and 0 for PR HEAD fafc784 in total

muraee · 2026-06-26T14:24:53Z

/retest-required

openshift-merge-bot · 2026-06-26T20:46:06Z

/hold

Revision fafc784 was retested 3 times: holding

celebdor · 2026-06-26T22:47:04Z

/override ci/prow/e2e-aws
The e2e-aws failure is unrelated to this PR's changes (etcd re-encryption backport). TestCreateClusterRequestServingIsolation test itself passed — the failure is purely in teardown (AWS resource cleanup deadline exceeded: 3 EC2 volumes + 1 NLB leaked). The same test passes on other release-4.22 PRs (e.g. #8851, #8725). All other e2e jobs (aks, azure-self-managed, kubevirt, upgrade, v2-aws) passed on the latest run.

celebdor · 2026-06-26T22:47:08Z

/hold cancel

openshift-ci · 2026-06-26T22:47:09Z

@celebdor: Overrode contexts on behalf of celebdor: ci/prow/e2e-aws

Details

In response to this:

/override ci/prow/e2e-aws
The e2e-aws failure is unrelated to this PR's changes (etcd re-encryption backport). TestCreateClusterRequestServingIsolation test itself passed — the failure is purely in teardown (AWS resource cleanup deadline exceeded: 3 EC2 volumes + 1 NLB leaked). The same test passes on other release-4.22 PRs (e.g. #8851, #8725). All other e2e jobs (aks, azure-self-managed, kubevirt, upgrade, v2-aws) passed on the latest run.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2026-06-26T23:08:25Z

@muraee: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws	`fafc784`	link	true	`/test e2e-aws`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

….0-4.22.3 Add azure CPO image overrides for 4.22 to backport API-driven Azure topology and private connectivity (CNTRLPLANE-3619, PR openshift#8721) and etcd data re-encryption after encryption key rotation (CNTRLPLANE-3656, PR openshift#8790). 4.22.4 does not need an override: both PRs merged 2026-06-26, before the 4.22.4 development cutoff (2026-07-01). - 4.22.0-4.22.3: CNTRLPLANE-3619 (PR openshift#8721), CNTRLPLANE-3656 (PR openshift#8790) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-ci Bot added the do-not-merge/needs-area label Jun 22, 2026

openshift-ci Bot requested review from cblecker and csrwng June 22, 2026 10:22

muraee force-pushed the backport-etcd-reencyrption-4.22 branch from 47cda15 to fafc784 Compare June 22, 2026 12:08

celebdor changed the title ~~[release-4.22] feat: add etcd data re-encryption after encryption key rotation (#8219)~~ CNTRLPLANE-3661: add etcd data re-encryption after encryption key rotation (#8219) Jun 22, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 22, 2026

muraee changed the title ~~CNTRLPLANE-3661: add etcd data re-encryption after encryption key rotation (#8219)~~ [release-4.22] CNTRLPLANE-3661: add etcd data re-encryption after encryption key rotation (#8219) Jun 22, 2026

celebdor changed the title ~~[release-4.22] CNTRLPLANE-3661: add etcd data re-encryption after encryption key rotation (#8219)~~ CNTRLPLANE-3661: add etcd data re-encryption after encryption key rotation (#8219) Jun 22, 2026

celebdor added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Jun 22, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 23, 2026

machi1990 mentioned this pull request Jun 25, 2026

test: add E2E test for AzureKmsKey update Azure/ARO-HCP#5746

Open

12 tasks

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jun 25, 2026

csrwng added backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. lgtm Indicates that a PR is ready to be merged. labels Jun 25, 2026

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 26, 2026

openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 26, 2026

openshift-merge-bot Bot merged commit 4b28599 into openshift:release-4.22 Jun 26, 2026
37 checks passed

celebdor mentioned this pull request Jul 2, 2026

CNTRLPLANE-3619: add azure CPO overrides for 4.22.0-4.22.3 #8908

Open

4 tasks

Uh oh!

Conversation

muraee commented Jun 22, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 22, 2026

Uh oh!

coderabbitai Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

codecov Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

openshift-ci-robot commented Jun 22, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoelSpeed commented Jun 23, 2026

Uh oh!

openshift-ci Bot commented Jun 23, 2026

Uh oh!

ahitacat commented Jun 25, 2026

Uh oh!

openshift-ci-robot commented Jun 25, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 25, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 25, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 26, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jun 26, 2026

AI Test Failure Analysis

Uh oh!

openshift-merge-bot Bot commented Jun 26, 2026

Uh oh!

muraee commented Jun 26, 2026

Uh oh!

openshift-merge-bot Bot commented Jun 26, 2026

Uh oh!

celebdor commented Jun 26, 2026

Uh oh!

celebdor commented Jun 26, 2026

Uh oh!

openshift-ci Bot commented Jun 26, 2026

Uh oh!

openshift-ci Bot commented Jun 26, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

coderabbitai Bot commented Jun 22, 2026 •

edited

Loading

codecov Bot commented Jun 22, 2026 •

edited

Loading

openshift-ci-robot commented Jun 22, 2026 •

edited by openshift-ci Bot

Loading