Skip to content

CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification#2029

Open
flavianmissi wants to merge 2 commits into
openshift:masterfrom
flavianmissi:image-verification-and-certification
Open

CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification#2029
flavianmissi wants to merge 2 commits into
openshift:masterfrom
flavianmissi:image-verification-and-certification

Conversation

@flavianmissi

Copy link
Copy Markdown
Member

No description provided.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label May 29, 2026
@openshift-ci-robot

openshift-ci-robot commented May 29, 2026

Copy link
Copy Markdown

@flavianmissi: This pull request references CNTRLPLANE-2938 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@flavianmissi flavianmissi changed the title CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification WIP CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification May 29, 2026
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 29, 2026
@flavianmissi flavianmissi changed the title WIP CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification [WIP] CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification May 29, 2026
@openshift-ci openshift-ci Bot requested review from dustymabe and enxebre May 29, 2026 14:14
@openshift-ci

openshift-ci Bot commented May 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jwmatthews for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Comment thread enhancements/kube-apiserver/kms-encryption-foundations.md Outdated
- Encrypt/decrypt operations under various conditions
- Plugin behavior during upgrades and migrations

Vendors extract the test binary from the release payload and execute it against

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add details about how we will prohibit the usage of arbitrary images?.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if there's an effective way to prevent this. We can only make it easier for customers to do the right thing, but as far as I know there aren't any hard checks we can do without hard-coding things in OpenShift's payload (which I don't think we should do).

What did you have in mind?

`ClusterImagePolicy` example:
```yaml
apiVersion: config.openshift.io/v1alpha1
kind: ClusterImagePolicy

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, this is an elegant way of fixing our problem.

@flavianmissi flavianmissi force-pushed the image-verification-and-certification branch 6 times, most recently from 87660ad to 9d7d987 Compare June 9, 2026 13:56
@flavianmissi flavianmissi force-pushed the image-verification-and-certification branch from 9d7d987 to 0ea7719 Compare June 9, 2026 14:00
@flavianmissi flavianmissi force-pushed the image-verification-and-certification branch from 0ea7719 to b0d2180 Compare June 18, 2026 13:17
@flavianmissi

flavianmissi commented Jun 18, 2026

Copy link
Copy Markdown
Member Author

/retitle CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification

@openshift-ci openshift-ci Bot changed the title [WIP] CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification CNTRLPLANE-2938: update kms EP with plugin image certification and signature verification Jun 18, 2026
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 18, 2026
@openshift-ci

openshift-ci Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

@flavianmissi: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Users must create a `ClusterImagePolicy` resource to configure OpenShift to
verify the image signature every time CRI-O pulls a plugin image. Users should
obtain the vendor's public key from the vendor's official documentation (e.g.,
HashiCorp's documentation for the Vault KMS plugin).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be great if we could prevent users from using arbitrary images.

This might be very difficult... unless we take ownership for shipping the ClusterImagePolicy resource with the right set of approved public key(s).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we don't create a key in key-controller, unless there is a respective ClusterImagePolicy resource for the given image?.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thing with shipping a ClusterImagePolicy in the payload is that we would need to hardcode the public key the vendor used to sign the plugin image. If they change the key between plugin versions, the key in openshift's payload would also need to change.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we don't create a key in key-controller, unless there is a respective ClusterImagePolicy resource for the given image?.

How are we going to detect a ClusterImagePolicy? We could document a label users could use? We must keep in mind that vendors might update the key they use to sign their images, in which at some point more than one ClusterImagePolicy will exist in the cluster for the same repository.

@flavianmissi flavianmissi Jun 19, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have a precedent for shipping a ClusterImagePolicy: https://github.com/openshift/cluster-update-keys/blob/main/manifests.rhel/0000_90_openshift-cluster-image-policy.yaml

Yes, the difference is that we (red hat) don't control the keys used to sign kms plugin images, while the keys in that repository are all controlled by us (red hat).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd be great if we could prevent users from using arbitrary images.

This is a very good point. It largely depends on the actual functional requirement.

It seems that when no policy is defined, no verification is performed and we end up running unverified images.

If a policy is defined, it may contain an untrusted key used for verification, which leads to the same outcome: running unverified images.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with image verification is that we need a public key from the image signer (vendor, i.e. vault) to verify the image. Unless we're willing to hard-code this public key in OpenShift's payload, there's no way to effectively block use of arbitrary images.

This challenge is inherent to the problem of not delivering images in the release payload. There's nothing we can do without hard-coding something in the payload (i.e image reference, public key, etc.).

I can't think it's a good idea to hard-code something we have no control over the lifecycle.

Comment thread enhancements/kube-apiserver/kms-encryption-foundations.md
Users must create a `ClusterImagePolicy` resource to configure OpenShift to
verify the image signature every time CRI-O pulls a plugin image. Users should
obtain the vendor's public key from the vendor's official documentation (e.g.,
HashiCorp's documentation for the Vault KMS plugin).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we don't create a key in key-controller, unless there is a respective ClusterImagePolicy resource for the given image?.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants