No NodeAffinity on PersistentVolumes when using Topology

[bh-tt]

Reopen of #566, reopened to make sure my comment is visible.

This is an issue for us as well, let me try to explain:

• we have a subset of all nodes that can reach the IBM storage (due to hardware networking requirements, some nodes cannot reach it)
• we have defined the topology awareness as specified in https://www.ibm.com/docs/en/stg-block-csi-driver/1.13.0?topic=configuring-csi-topology, with nodes capable of accessing the storage array labeled in this manner, with a supported_topologies key in the storage array secret used by the StorageClass. The CSINode resource confirms that this is picked up
• now, we create a PVC with this StorageClass and a Pod using this PVC. Note that we do not explicitly run the Pod with a nodeselector for nodes that can reach the storage
• if the pod is scheduled on one of the nodes that can reach the array, the PVC is created, the pod starts, all is OK. If the pod is scheduled on a node that cannot reach the array, it remains in pending state. This is annoying, but not the real issue
• now, the pod is rescheduled on a node that cannot reach the array, and it remains pending forever.

The k8s devs have found a way to fix this: the PV can have a nodeSelector, as @dje4om noted in his last comment, allowing the k8s scheduler to see 'hey, this pod needs a PV that only accessible from certain nodes. I'll only consider nodes that have this label'.

I think the disconnect here is that the StorageClass/secret/node labeling is very much an cluster administrator concern, whereas app developers only see a StorageClass name they can use. App developers do not (and should not) need to know about scheduling constraints that apply to a CSI storage type. The current 'solution' in this project is to label pods with nodeselectors related to the storage the pod uses, which means app developers need much more info from cluster administrators while a better and simpler solution exists: filling the PV NodeSelector, which can be done by simply adding the actually selected topology to the GRPC CreateVolumeResponse.

In go, that'd work like so:

	return &csi.CreateVolumeResponse{
		Volume: &csi.Volume{
			VolumeId:      fs.id.String(),
			CapacityBytes: fs.capacity,
			AccessibleTopology: []*csi.Topology{
				<1 or more topologies that can access the given volume, for the simple case this would be equal to the result of https://github.com/IBM/ibm-block-csi-driver/blob/926727bdb59a32d96700bf9b02df7d75a4096fc0/controllers/servers/utils.py#L53>
			},
			VolumeContext: req.Parameters}}, nil