How to verify currently used DRBD protocol

[CJLvU]

I have a use case where data integrity is not as important as performance and want to use DRBD protocol A to eliminate or at least minimize replication overhead. I'm looking for a way to verify that the currently used DRBD protocol is actually set to A since latency and consequently performance is way below expectations for a bunch of VMs running on NVMe backed ZFS with only a linux bridge between them.

I have tried drbdsetup show in one of the linstor satellite pods but I'm not seeing anything regarding the DRBD protocol:

resource "pvc-fe585ec9-ec00-4bf2-a222-f82eec058a5e" {
    options {
        on-no-data-accessible   suspend-io;
        quorum                  majority;
        on-suspended-primary-outdated   force-secondary;
    }
    _this_host {
        node-id                 1;
        volume 0 {
            device                      minor 1001;
            disk                        "/dev/vg1-thin/pvc-fe585ec9-ec00-4bf2-a222-f82eec058a5e_00000";
            meta-disk                   internal;
            disk {
                rs-discard-granularity  65536; # bytes
            }
        }
    }
    connection {
        _peer_node_id 0;
        path {
            _this_host ipv4 10.244.0.35:7001;
            _remote_host ipv4 10.244.3.35:7001;
        }
        _cstate Connected;
        net {
            cram-hmac-alg       "sha1";
            shared-secret       "R6513b/Jz7NN5zKDDwMA";
            rr-conflict         retry-connect;
            verify-alg          "sha512";
            _name               "vm02-1gpu";
        }
    }
    connection {
        _peer_node_id 2;
        path {
            _this_host ipv4 10.244.0.35:7001;
            _remote_host ipv4 10.244.2.37:7001;
        }
        _cstate Connected;
        net {
            cram-hmac-alg       "sha1";
            shared-secret       "R6513b/Jz7NN5zKDDwMA";
            rr-conflict         retry-connect;
            verify-alg          "sha512";
            _name               "vm05";
        }
    }
}

Is there a way to get that information from the Linstor cluster?

[WanzenBug]

It should be visible in drbdsetup status. How did you set it currently?

[CJLvU]

It's specified in the LinstorNodeConnections section of the linstor-cluster helm-chart values.yaml that is deployed as a ConfigMap by FluxCD:

apiVersion: v1
kind: ConfigMap
metadata:
  name: piraeus-datastore-values
  namespace: piraeus-datastore
data:
  values-yaml: |
    linstorCluster: {}
    linstorPassphrase: ""
    createApiTLS: ""
    createInternalTLS: ""
    ingress:
      enabled: false
      ingressClassName: ""
      labels: {}
      annotations: {}
      host: linbit-sds.local
      path: /
      pathType: ImplementationSpecific
      tls: []
    linstorSatelliteConfigurations:
      - name: satellite-configuration-a
        namespace: piraeus-datastore
        storagePools:
          - name: lvm-thin
            lvmThinPool:
              volumeGroup: vg1-thin
              thinPool: thin
            source:
              hostDevices:
                # - /dev/nvme0n1
                - /dev/disk/by-id/nvme-QEMU_NVMe_Ctrl_incus_DISK01
        podTemplate:
          spec:
            initContainers:
              - name: drbd-shutdown-guard
                $patch: delete
              - name: drbd-module-loader
                $patch: delete
            volumes:
              - name: run-systemd-system
                $patch: delete
              - name: run-drbd-shutdown-guard
                $patch: delete
              - name: systemd-bus-socket
                $patch: delete
              - name: lib-modules
                $patch: delete
              - name: usr-src
                $patch: delete
              - name: etc-lvm-backup
                hostPath:
                  path: /var/etc/lvm/backup
                  type: DirectoryOrCreate
              - name: etc-lvm-archive
                hostPath:
                  path: /var/etc/lvm/archive
                  type: DirectoryOrCreate
    linstorNodeConnections:
      - name: connection-a
        properties:
          - name: DrbdOptions/Net/protocol
            value: A
    monitoring:
      enabled: false
      dashboard:
        enabled: true
    storageClasses:
      - name: linstor-local
        annotations:
          storageclass.kubernetes.io/is-default-class: "false"
        labels:
          example.com/storage-class: "true"
        reclaimPolicy: Delete
        allowVolumeExpansion: true
        volumeBindingMode: WaitForFirstConsumer
        provisioner: linstor.csi.linbit.com
        parameters:
          linstor.csi.linbit.com/autoPlace: "1"
          linstor.csi.linbit.com/storagePool: lvm-thin
          linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"
      - name: linstor-replica3
        annotations:
          storageclass.kubernetes.io/is-default-class: "true"
        labels:
          example.com/storage-class: "true"
        reclaimPolicy: Delete
        allowVolumeExpansion: true
        volumeBindingMode: WaitForFirstConsumer
        provisioner: linstor.csi.linbit.com
        parameters:
          linstor.csi.linbit.com/autoPlace: "3"
          linstor.csi.linbit.com/storagePool: lvm-thin
          linstor.csi.linbit.com/allowRemoteVolumeAccess: "false"
    volumeSnapshotClasses: []
    volumeGroupSnapshotClasses: []

I briefly suspected that the lack of selectors means that the connection settings are not being used but the docs say that default behavior is to apply it to all nodes.

[WanzenBug]

Hmm, that should be correct. You can check the output of kubectl exec deploy/linstor-controller -- linstor nc l and see if it shows the node connection information. You can also check kubectl describe linstornodeconnection and see if there are any errors.

[CJLvU]

kubectl describe linstornodeconnection and kubectl exec -n piraeus-datastore deploy/linstor-controller -- linstor nc list-properties vm04 vm05 both show protocol A as configured without any errors. The Properties column of the ascii table cuts off any useful info when using kubectl exec deploy/linstor-controller -- linstor nc l unfortunately.

All of this points towards Protocol A being used but sequential write performance tests using fio nonetheless lags behind sequential reads by a factor of 30x which seems extreme. I'll mark this as answered but if you have any additional pointers I'd be happy to check them against my setup.