This project is installed using the operator pattern: you deploy an operator and (instances of) some CRDs, and the operator creates the underlying daemonsets and controllers. We've run into a few issues with this:
- it is hard to configure specific settings of the underlying daemonset/controller, for example priorityclassname. Each of these changes needs a change in the operator project
- this subsequently creates a high maintenance burden for the operator
- since the IBMBlockCSI and HostDefiner are not well-known resources, update tooling such as renovate fails to recognize these items as images and so does not automatically create updates. It also falls to the users of this project to figure out which versions of the csi sidecar containers should be used
Many of these issues become a lot simpler if this project would release installation yaml files for the driver and the hostdefiner, containing the deployment/daemonset/statefulset, serviceaccount, clusterroles and clusterrolebindings and the CRDs used. This allows users to use Kustomize to update the yamls to their liking while being much easier for you to maintain.
In order to setup priorityclassnames for this project, we've already done this for the daemonset and controller, resulting in no longer needing (a very high privileged) operator deployment. The only thing you would need to do is add the CRDs and setup a small bit of automation to release these yamls with a github release. Then users can install this using either a kubectl apply https://github.com/IBM/ibm-block-csi-driver/releases/v1.12.5/download/hostdefiner.yaml or by including these files in a kustomization.yaml.
hostdefiner.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/component: hostdefiner
app.kubernetes.io/instance: host-definer
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
name: hostdefiner
namespace: ibm-block-csi-operator
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/component: hostdefiner
template:
metadata:
labels:
app.kubernetes.io/component: hostdefiner
app.kubernetes.io/instance: host-definer
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
spec:
priorityClassName: it-infrastructure
containers:
- env:
- name: PREFIX
- name: CONNECTIVITY_TYPE
value: iscsi
- name: ALLOW_DELETE
value: "true"
- name: DYNAMIC_NODE_LABELING
value: "true"
- name: PORT_SET
image: quay.io/ibmcsiblock/ibm-block-csi-host-definer:1.12.5
imagePullPolicy: IfNotPresent
name: ibm-block-csi-host-definer
resources:
limits:
cpu: 800m
memory: 400Mi
requests:
cpu: 40m
memory: 40Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
serviceAccountName: hostdefiner
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: host-definer
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
name: hostdefiner
namespace: ibm-block-csi-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ibm:hostdefiner
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- list
- apiGroups:
- ""
resources:
- events
verbs:
- create
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
- apiGroups:
- storage.k8s.io
resources:
- csinodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- patch
- watch
- list
- apiGroups:
- csi.ibm.com
resources:
- hostdefiners
verbs:
- get
- list
- watch
- apiGroups:
- csi.ibm.com
resources:
- hostdefinitions
verbs:
- '*'
- apiGroups:
- csi.ibm.com
resources:
- hostdefinitions/status
verbs:
- '*'
- apiGroups:
- apps
resources:
- daemonsets
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ibm:hostdefiner
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm:hostdefiner
subjects:
- kind: ServiceAccount
name: hostdefiner
namespace: ibm-block-csi-operatorapiVersion: apps/v1
kind: DaemonSet
metadata:
labels:
app.kubernetes.io/component: csi-node
app.kubernetes.io/instance: ibm-block-csi
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
name: ibm-block-csi-node
namespace: ibm-block-csi-operator
spec:
selector:
matchLabels:
app.kubernetes.io/component: csi-node
template:
metadata:
annotations:
productID: ibm-block-csi-driver
productName: ibm-block-csi-driver
labels:
app.kubernetes.io/component: csi-node
app.kubernetes.io/instance: ibm-block-csi
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
spec:
priorityClassName: system-cluster-critical
containers:
- args:
- --csi-endpoint=$(CSI_ENDPOINT)
- --hostname=$(KUBE_NODE_NAME)
- --config-file-path=./config.yaml
- --loglevel=$(CSI_LOGLEVEL)
- --max-invocations=2
- --clean-scsi-device=true
env:
- name: CSI_ENDPOINT
value: unix:///csi/csi.sock
- name: CSI_LOGLEVEL
value: info
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
image: quay.io/ibmcsiblock/ibm-block-csi-driver-node:1.12.5
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
httpGet:
path: /healthz
port: 9808
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 3
name: ibm-block-csi-node
ports:
- containerPort: 9808
name: healthz
protocol: TCP
resources:
limits:
cpu: "1"
memory: 500Mi
requests:
cpu: 40m
memory: 40Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- CHOWN
- FSETID
- FOWNER
- SETGID
- SETUID
- DAC_OVERRIDE
drop:
- ALL
privileged: true
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /var/lib/kubelet/pods
mountPropagation: Bidirectional
name: mountpoint-dir
- mountPath: /dev
name: device-dir
- mountPath: /sys
name: sys-dir
- mountPath: /host
mountPropagation: Bidirectional
name: host-dir
- mountPath: /etc/iscsi
name: iscsi
- args:
- --csi-address=$(ADDRESS)
- --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
- --v=5
env:
- name: ADDRESS
value: /csi/csi.sock
- name: DRIVER_REG_SOCK_PATH
value: /var/lib/kubelet/plugins/block.csi.ibm.com/csi.sock
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.15.0@sha256:11f199f6bec47403b03cb49c79a41f445884b213b382582a60710b8c6fdc316a
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- rm -rf /registration/ibm-block-csi-driver-reg.sock /csi/csi.sock
name: csi-node-driver-registrar
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /csi
name: socket-dir
- mountPath: /registration
name: registration-dir
- args:
- --csi-address=/csi/csi.sock
- --health-port=9808
image: registry.k8s.io/sig-storage/livenessprobe:v2.17.0@sha256:9b75b9ade162136291d5e8f13a1dfc3dec71ee61419b1bfc112e0796ff8a6aa9
imagePullPolicy: IfNotPresent
name: livenessprobe
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /csi
name: socket-dir
hostIPC: true
hostNetwork: true
serviceAccountName: ibm-block-csi-node-sa
volumes:
- hostPath:
path: /var/lib/kubelet/pods
type: Directory
name: mountpoint-dir
- hostPath:
path: /var/lib/kubelet/plugins/block.csi.ibm.com
type: DirectoryOrCreate
name: socket-dir
- hostPath:
path: /var/lib/kubelet/plugins_registry
type: Directory
name: registration-dir
- hostPath:
path: /dev
type: Directory
name: device-dir
- hostPath:
path: /sys
type: Directory
name: sys-dir
- hostPath:
path: /
type: Directory
name: host-dir
- hostPath:
path: /etc/iscsi
type: Directory
name: iscsi
updateStrategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 3
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
productID: ibm-block-csi-driver
productName: ibm-block-csi-driver
labels:
app.kubernetes.io/component: csi-controller
app.kubernetes.io/instance: ibm-block-csi
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
name: ibm-block-csi-controller
namespace: ibm-block-csi-operator
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: csi-controller
serviceName: ibm-block-csi-controller
template:
metadata:
annotations:
productID: ibm-block-csi-driver
productName: ibm-block-csi-driver
labels:
app.kubernetes.io/component: csi-controller
app.kubernetes.io/instance: ibm-block-csi
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
spec:
priorityClassName: system-cluster-critical
containers:
- args:
- --csi-endpoint=$(CSI_ENDPOINT)
env:
- name: CSI_ENDPOINT
value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
- name: CSI_LOGLEVEL
value: INFO
- name: ENABLE_CALL_HOME
value: "true"
- name: ODF_VERSION_FOR_CALL_HOME
- name: SVC_SSH_PORT
value: "22"
image: quay.io/ibmcsiblock/ibm-block-csi-driver-controller:1.12.5
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 30
httpGet:
path: /healthz
port: 9808
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 100
name: ibm-block-csi-controller
ports:
- containerPort: 9808
name: healthz
protocol: TCP
resources:
limits:
cpu: 800m
memory: 400Mi
requests:
cpu: 40m
memory: 40Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --csi-address=$(ADDRESS)
- --v=5
- --timeout=120s
- --default-fstype=ext4
- --worker-threads=16
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: registry.k8s.io/sig-storage/csi-provisioner:v5.3.0@sha256:bb057f866177d5f4139a1527e594499cbe0feeb67b63aaca8679dfdf0a6016f9
imagePullPolicy: IfNotPresent
name: csi-provisioner
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --csi-address=$(ADDRESS)
- --v=5
- --timeout=180s
- --worker-threads=16
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: registry.k8s.io/sig-storage/csi-attacher:v4.10.0@sha256:be59d0556508d3dd419cc34c53062f170a28902ef36e832b947c7796458a083d
imagePullPolicy: IfNotPresent
name: csi-attacher
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --csi-address=$(ADDRESS)
- --v=5
- --timeout=120s
- --worker-threads=16
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: registry.k8s.io/sig-storage/csi-snapshotter:v8.4.0@sha256:c7e0a3718832b6197ce8b29fefb3fed3d84f4fbcdf08f4606140dbec2566501d
imagePullPolicy: IfNotPresent
name: csi-snapshotter
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --csi-address=$(ADDRESS)
- --v=5
- --timeout=30s
- --handle-volume-inuse-error=false
- --workers=16
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: registry.k8s.io/sig-storage/csi-resizer:v1.14.0@sha256:5e7cbb63fd497fa913caa21fee1a69f727c220c6fa83c5f8bb0995e2ad73a474
imagePullPolicy: IfNotPresent
name: csi-resizer
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --leader-election-namespace=ibm-block-csi-operator
- --driver-name=block.csi.ibm.com
- --csi-address=$(ADDRESS)
- --zap-log-level=5
- --rpc-timeout=30s
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: quay.io/ibmcsiblock/csi-block-volumereplication-operator:v0.9.0
imagePullPolicy: IfNotPresent
name: csi-addons-replicator
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --driver-name=block.csi.ibm.com
- --csi-address=$(ADDRESS)
- --rpc-timeout=30s
- --multiple-vgs-to-pvc=false
- --disable-delete-pvcs=true
env:
- name: ADDRESS
value: /var/lib/csi/sockets/pluginproxy/csi.sock
image: quay.io/ibmcsiblock/csi-volume-group-operator:v0.9.1
imagePullPolicy: IfNotPresent
name: csi-volume-group
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /var/lib/csi/sockets/pluginproxy/
name: socket-dir
- args:
- --csi-address=/csi/csi.sock
- --health-port=9808
image: registry.k8s.io/sig-storage/livenessprobe:v2.17.0@sha256:9b75b9ade162136291d5e8f13a1dfc3dec71ee61419b1bfc112e0796ff8a6aa9
imagePullPolicy: IfNotPresent
name: livenessprobe
resources:
limits:
cpu: 200m
memory: 200Mi
requests:
cpu: 20m
memory: 20Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
volumeMounts:
- mountPath: /csi
name: socket-dir
securityContext:
fsGroup: 9999
runAsUser: 9999
serviceAccountName: ibm-block-csi-controller-sa
volumes:
- emptyDir: {}
name: socket-dir
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: ibm-block-csi
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
name: ibm-block-csi-node-sa
namespace: ibm-block-csi-operator
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app.kubernetes.io/instance: ibm-block-csi
app.kubernetes.io/name: ibm-block-csi-driver
csi: ibm
name: ibm-block-csi-controller-sa
namespace: ibm-block-csi-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ibm-block-csi-external-attacher-clusterrole
rules:
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- watch
- patch
- apiGroups:
- storage.k8s.io
resources:
- csinodes
verbs:
- get
- list
- watch
- apiGroups:
- storage.k8s.io
resources:
- volumeattachments
verbs:
- get
- list
- watch
- patch
- apiGroups:
- storage.k8s.io
resources:
- volumeattachments/status
verbs:
- patch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ibm-block-csi-external-provisioner-clusterrole
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- watch
- create
- delete
- apiGroups:
- ""
resources:
- persistentvolumeclaims
verbs:
- get
- list
- watch
- update
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- list
- watch
- create
- update
- patch
- apiGroups:
- snapshot.storage.k8s.io
resources:
- volumesnapshots
verbs:
- get
- list
- apiGroups:
- snapshot.storage.k8s.io
resources:
- volumesnapshotcontents
verbs:
- get
- list
- apiGroups:
- storage.k8s.io
resources:
- csinodes
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- list
- watch
- apiGroups:
- storage.k8s.io
resources:
- volumeattachments
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ibm-block-csi-external-resizer-clusterrole
rules:
- apiGroups:
- ""
resources:
- persistentvolumes
verbs:
- get
- list
- watch
- patch
- apiGroups:
- ""
resources:
- persistentvolumeclaims
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- persistentvolumeclaims/status
verbs:
- patch
- apiGroups:
- ""
resources:
- events
verbs:
- list
- watch
- create
- update
- patch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ibm-block-csi-external-snapshotter-clusterrole
rules:
- apiGroups:
- ""
resources:
- events
verbs:
- list
- watch
- create
- update
- patch
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- list
- apiGroups:
- snapshot.storage.k8s.io
resources:
- volumesnapshotclasses
verbs:
- get
- list
- watch
- apiGroups:
- snapshot.storage.k8s.io
resources:
- volumesnapshotcontents
verbs:
- create
- get
- list
- watch
- update
- delete
- patch
- apiGroups:
- snapshot.storage.k8s.io
resources:
- volumesnapshotcontents/status
verbs:
- update
- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: ibm-block-csi-node-clusterrole
rules:
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ibm-block-csi-external-attacher-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-block-csi-external-attacher-clusterrole
subjects:
- kind: ServiceAccount
name: ibm-block-csi-controller-sa
namespace: ibm-block-csi-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ibm-block-csi-external-provisioner-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-block-csi-external-provisioner-clusterrole
subjects:
- kind: ServiceAccount
name: ibm-block-csi-controller-sa
namespace: ibm-block-csi-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ibm-block-csi-external-resizer-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-block-csi-external-resizer-clusterrole
subjects:
- kind: ServiceAccount
name: ibm-block-csi-controller-sa
namespace: ibm-block-csi-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ibm-block-csi-external-snapshotter-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-block-csi-external-snapshotter-clusterrole
subjects:
- kind: ServiceAccount
name: ibm-block-csi-controller-sa
namespace: ibm-block-csi-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: ibm-block-csi-node-clusterrolebinding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: ibm-block-csi-node-clusterrole
subjects:
- kind: ServiceAccount
name: ibm-block-csi-node-sa
namespace: ibm-block-csi-operator
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: block.csi.ibm.com
spec:
attachRequired: true
fsGroupPolicy: ReadWriteOnceWithFSType
podInfoOnMount: false
requiresRepublish: false
seLinuxMount: false
storageCapacity: false
volumeLifecycleModes:
- Persistent
This project is installed using the operator pattern: you deploy an operator and (instances of) some CRDs, and the operator creates the underlying daemonsets and controllers. We've run into a few issues with this:
Many of these issues become a lot simpler if this project would release installation yaml files for the driver and the hostdefiner, containing the deployment/daemonset/statefulset, serviceaccount, clusterroles and clusterrolebindings and the CRDs used. This allows users to use Kustomize to update the yamls to their liking while being much easier for you to maintain.
In order to setup priorityclassnames for this project, we've already done this for the daemonset and controller, resulting in no longer needing (a very high privileged) operator deployment. The only thing you would need to do is add the CRDs and setup a small bit of automation to release these yamls with a github release. Then users can install this using either a
kubectl apply https://github.com/IBM/ibm-block-csi-driver/releases/v1.12.5/download/hostdefiner.yamlor by including these files in a kustomization.yaml.hostdefiner.yaml: