Driver uses sg_utils host commands; breaks under debian trixie

[bh-tt]

We recently upgraded a cluster to debian trixie, which resulted in this CSI driver failing to mount any volumes, with a cryptic 'volume ID not found' message. We managed to trace the problem to the use of the sg_inq command (installed on the host) in the csi-node daemonset, whose default output changed with the trixie upgrade.

We applied the following patch to the driver to get it to work again (at least, mounting and unmounting worked again):

diff --git a/Dockerfile-csi-node b/Dockerfile-csi-node
index 0f3504d1..bf829c7b 100644
--- a/Dockerfile-csi-node
+++ b/Dockerfile-csi-node
@@ -69,6 +69,7 @@ RUN    ln -s /chroot/chroot-host-wrapper.sh /chroot/blkid \
     && ln -s /chroot/chroot-host-wrapper.sh /chroot/nvme \
     && ln -s /chroot/chroot-host-wrapper.sh /chroot/resize2fs \
     && ln -s /chroot/chroot-host-wrapper.sh /chroot/sg_inq \
+    && ln -s /chroot/chroot-host-wrapper.sh /chroot/sg_vpd \
     && ln -s /chroot/chroot-host-wrapper.sh /chroot/sg_map \
     && ln -s /chroot/chroot-host-wrapper.sh /chroot/umount \
     && ln -s /chroot/chroot-host-wrapper.sh /chroot/xfs_growfs
diff --git a/node/pkg/driver/device_connectivity/device_connectivity_helper_scsigeneric.go b/node/pkg/driver/device_connectivity/device_connectivity_helper_scsigeneric.go
index f41fe5cc..484b3697 100644
--- a/node/pkg/driver/device_connectivity/device_connectivity_helper_scsigeneric.go
+++ b/node/pkg/driver/device_connectivity/device_connectivity_helper_scsigeneric.go
@@ -581,13 +581,13 @@ func (o OsDeviceConnectivityHelperGeneric) GetWwnByScsiInq(dev string) (string,
                        associated with the target port
                          Relative target port: 0xd22
        */
-       sgInqCmd := "sg_inq"
+       sgInqCmd := "sg_vpd"
 
        if err := o.Executer.IsExecutable(sgInqCmd); err != nil {
                return "", err
        }
 
-       args := []string{"-p", "0x83", dev}
+       args := []string{"--long", "-p", "0x83", dev}
        // add timeout in case the call never comes back.
        logger.Debugf("Calling [%s] with timeout", sgInqCmd)
        outputBytes, err := o.Executer.ExecuteWithTimeout(TimeOutSgInqCmd, sgInqCmd, args)

Versions:
old:

sg_inq --version
Version string: 2.10 20210328
$ sg_vpd --version
version: 1.63 20210328

new:

$ sg_inq --version
Version string: 2.48 20230606
$ sg_vpd --version
version: 1.96 20230622

However, we probably cannot apply this patch to the driver as a whole, since this is dependent on which version of sg_utils is installed on the kubernetes node.

I see a few paths forward:

• detect the version of sg_utils on the host, and switch the command based on that -> how to detect the version robustly?
• consider installing sg_utils and the other host commands used into the image, so this driver can function and upgrade these packages independent of the host

I think most CSI drivers choose the second option.

[kbolapon]

Same issue with OKD 4.20 since there it is Centos coreos 10 so ibm-block-csi does not work.

[kbolapon]

I know that Redhat Openshift still use coreos 9.6 which contains sg_utils 1.47 which works with ibm-block-csi but newer OKD 4.20 comes with centos 10 which is based on Redhat 10 containing sg_utils 1.48 which breaks ibm-block-csi.

Is there any work regarding to make ibm-block-csi compatible with sg_utils 1.48 or is only solution to ditch Storewize and reformat all the drives and use CEPH instead?

[kbolapon]

Do I understand correct now that Redhat Openshift stable 4.21 and 4.22 use rhel coreos 10 which also contains the sg3_utils-1.48 which breaks sg_inq?

I do not feel for test an upgrade just to see if it does not work.

[shlomitn]

Sorry, I missed this.
What you described will be resolved in 1.14.0. In the meantime, there may be workarounds this. I know for SUSe we found some workaround.
About OCP 4.21 I can report that we did test this with our CSI 1.13.1 and it worked, so I believe it should be ok.

[wpatton]

Sorry, I missed this. What you described will be resolved in 1.14.0. In the meantime, there may be workarounds this. I know for SUSe we found some workaround. About OCP 4.21 I can report that we did test this with our CSI 1.13.1 and it worked, so I believe it should be ok.

Can you share the workaround you found for SUSE?