This guide describes the process for promoting an API in Crossplane from one version to the next. This process should be handled with care over multiple releases, as it has proven fairly easy for Crossplane's API promotions of the past to be problematic for both upgrades and downgrades, as demonstrated by #6148, #5932, and #4400.
An explanation of Crossplane's feature lifecycle can be found in the Feature Lifecycle docs page.
The root cause we have encountered in past problematic API promotions is that we try to drop a particular version from a CRD while there may still be resources of that version stored in etcd. Kubernetes does not allow this action because it can result in possible data loss.
Note that this can happen on both upgrades as well as downgrades. Let's briefly summarize one common downgrade path where we've encountered this issue:
- A Crossplane API is promoted to Beta and the storage version for the CRD is updated to Beta in the same release
- User installs this new Crossplane version and creates a Beta resource that is stored in etcd as Beta
- User downgrades to the previous Crossplane version where the CRD's Beta version did not exist
- The Crossplane init container tries to update the CRD back to the Alpha version, which drops the Beta version
- The Kubernetes API server rejects this action because there might still be
Beta resources stored in etcd according to the CRD's
status.storedVersionsfield - Crossplane init container crashes and the Crossplane pod restarts in a loop
There are a few simple principles we can follow in order to avoid this common scenario when promoting Crossplane APIs:
- Never introduce or drop a CRD version while also bumping the storage version in the same release
- Always migrate resources to the current storage version during Crossplane initialization
Adherence to these two principles should result in never dropping a CRD version that still has resources of that version stored in etcd, for both the upgrade and downgrade directions.
The following table outlines the steps to introduce and then promote an API safely across multiple versions while maintaining safe upgrades and downgrades:
| Version | Description | Alpha | Beta | Migration |
|---|---|---|---|---|
| v0.1 | introduce new API as alpha | storage, served | not exist | none |
| v0.2 | promote to beta | storage, served | served | migrate to alpha |
| v0.3 | bump storage to beta | served | storage, served | migrate to beta |
| v0.4 | drop alpha | not exist | storage, served | none |
The same process could be applied again when promoting the API from Beta to v1 GA.
This section contains some helpful pointers to areas of the codebase involved in promoting an API. These are just general direction and not entirely prescriptive, because you'll need to make specific decisions for your promotion based on the workflow defined above.
- Define the API types under the
apisdirectory, for exampleapis/apiextensions/v1beta1 - Set the storage version
+kubebuilder:storageversionmarker on the correct version, e.g., as shown here for Usage v1alpha1 - Duplicate the API to other versions if needed using
generate.go, also instructing the duplicate script to set the storage version if needed - Run
earthly +generateto generate the CRDs and check them for sanity in thecluster/crdsdirectory - Include a migrator if needed to ensure resources are migrated to the
current storage version and the CRD status is updated to declare that it only
has resources stored in etcd of the current storage version
- Note that the version passed into the migrator is the "old" version you want to migrate from, not the target version you want to migrate to
- Update the feature flag for this API to reflect its new maturity level
- Update the feature flag docs to reflect the new maturity level there as well