Upgrade from stable/1.31 to stable/1.32 ends with waiting status ('422 Unprocessable Entity')

[aborsik]

Hello folks,

I recently upgraded two Juju charms managed k8s clusters from 1.31 to 1.32. After all other units were upgraded then I upgraded ceph-csi using:

juju refresh ceph-csi --channel=1.32/stable (previously it was 1.31/stable and version was set to v3.11.0)

I have two k8s control-plane units (kubernetes-control-plane/0 and /1), k8s units and all other Juju model units are ok including one of ceph-csi unit but first/leader ceph-csi unit is constantly in waiting status:

Failed Applying Deployment/default/csi-cephfsplugin-provisioner -> Deployment.apps "csi-cephfsplugin-provisioner" is ...

I already tried to switch to most recent version:

juju run ceph-csi/leader list-versions
juju config ceph-csi release=v3.13.0

But it could not upgrade deployment on k8s - please see attached Juju debug log. Basically it ends with ERROR unit.ceph-csi/1.juju-log Failed Applying Deployment/default/csi-cephfsplugin-provisioner with httpx.HTTPStatusError: Client error '422 Unprocessable Entity'

What I'm doing wrong?

ceph-csi-juju-debug-log.txt

[marianhromiak]

Hi, I'm @aborsik's colleague. We’ve identified the root cause of this upgrade failure.

Root Cause
The issue stems from a configuration conflict during the upgrade reconciliation. In our environment, the metrics ports were disabled (set to -1)::

$ for key in metrics-port-cephfsplugin metrics-port-cephfsplugin-provisioner metrics-port-rbdplugin metrics-port-rbdplugin-provisioner; do juju config ceph-csi ${key}; done
-1
-1
-1
-1

However, the existing 3.11.0 deployment already contained liveness-prometheus sidecars.

When the operator attempts to upgrade to 3.13.0, the manifest filtering logic excludes these containers from the desired state because the ports are -1. This results in the operator trying to apply an invalid/empty patch to the existing containers in the csi-cephfsplugin-provisioner, csi-rbdplugin-provisioner, csi-cephfsplugin, and csi-rbdplugin resources.

Workaround

We resolved the "blocked" state by manually aligning the cluster state with the configuration before re-attempting the upgrade:

1. Roll back the charm version to 3.11.0.
2. Manually delete the `liveness-prometheus` container block from the manifests of the affected Deployments and DaemonSets.
3. Perform the upgrade to `3.13.0` again.

Once the "orphaned" sidecar containers were removed manually, the operator was able to complete the upgrade successfully.

Deleted container:

  - args:
    - --type=liveness
    - --endpoint=$(CSI_ENDPOINT)
    - --metricsport=8681
    - --metricspath=/metrics
    - --polltime=60s
    - --timeout=3s
    env:
    - name: CSI_ENDPOINT
      value: unix:///csi/csi-provisioner.sock
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: status.podIP
    image: rocks.canonical.com:443/cdk/cephcsi/cephcsi:v3.11.0
    imagePullPolicy: IfNotPresent
    name: liveness-prometheus
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /csi
      name: socket-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-xs5mv
      readOnly: true