csi-nfs-server reports a few failed jobs with 'failed with result 'dependency''

[utegental]

From what I see, looks like it reports some mount-related errors. Yet, service is not going down.

[WanzenBug]

Can you exec into the container and run journalctl and check the error message for the mount-recovery volume?

Note that it is expected that the service is still up. The daemonset implements a internal failover strategy on a per-volume basis, so that we do not need one set of exporter pods per volume, which would quickly exhaust the available Pods if using many RWX volumes.

[utegental]

csi-nfs-server.log

[WanzenBug]

Whats the output of you run drbdsetup status -vvv in that container. It looks like DRBD reactor thinks the services should be started, but at the same time DRBD thinks the resources do not exist.

Which is weird, because drbd-reactor only starts resources that have been reported as ready by DRBD.

[bernardgut]

EDIT : KEEPING THIS HERE FOR ARCHIVE PURPOSES AS A REMINDER NOT TO TRUST YOUR LLM TOO MUCH ON DEBUG TASKS:

Reliable reproducer + recovery for this issue.

Reproducer

1. Provision an RWX PVC on a SC using the linstor-csi NFS helper.
2. Start a pod that mounts it.
3. Delete the consumer pod. CSI issues NodeUnpublish/NodeUnstage; drbd-reactor demotes the resource; the templated unit chain (drbd-promote@, mount-export@, nfs-ganesha@, advertise-nfs-endpoint@) deactivates.
4. Create a new consumer pod for the same PVC within the same hour.
5. New pod hangs in ContainerCreating with MountVolume.SetUp failed... rpc error: code = DeadlineExceeded.

Pre-condition: drbd-reactor's templated units must have transitioned through a deactivation cycle at least once during the helper pod's current lifetime.

Broken state

Inside the linstor-csi-nfs-server pod on the diskful node:

drbd-promote@pvc-XXXX.service: Skipped due to 'exec-condition'.
prepare-device-links@pvc-XXXX.service: Bound to unit drbd-promote@..., but unit isn't active.
drbd-services@pvc-XXXX.target: start failed with result 'dependency'.
(... entire chain in 'dependency' failure ...)

LINSTOR and DRBD both report the resource healthy at the same time: Unused / Ok / UpToDate, role:Secondary disk:UpToDate connection:Ok, no StandAlone peer. The resource is genuinely promotable; drbd-reactor declines.

systemctl reload drbd-reactor.service does not recover. Reload logs as successful, but the per-resource units remain in skipped/failed state.

Workaround (no data loss)

Delete the helper pod on the affected node: kubectl -n piraeus-datastore delete pod <linstor-csi-nfs-server-on-affected-node>. DaemonSet recreates it, drbd-reactor re-evaluates the exec-condition with the pending mount request, the chain starts in order, and the consumer pod's ContainerCreating clears within ~30s. No DRBD reattach, no resync.

Things that do NOT recover the state: systemctl reload drbd-reactor, host reboot (same skip recurs on first activation post-boot), linstor resource toggle-disk.

Hypothesis

ExecCondition= is evaluated once at unit-template-instantiation time and is sticky thereafter. When drbd-reactor instantiates the chain in response to a DRBD events2 message at T, the exec-condition observes the current state. If NodePublishVolume arrives at T+dt, the unit is already terminally Skipped and systemd does not re-evaluate.

This aligns with #415's architectural observation that the controller-generated ConfigMap can desynchronize from LINSTOR's actual state.

Suggested next step

Replace or supplement ExecCondition= with ConditionPathExists=/run/drbd/by-res/%i (or equivalent) on the drbd-promote@ template. Systemd re-checks Condition*= on every Start request, whereas ExecCondition= does not.

Environment

piraeus-operator v2.10.5, linstor-csi v1.10.6, piraeus-server v1.33.1, drbd-reactor v1.11.0, DRBD 9.3.1, Talos 1.13.0 / kernel 6.18.24. RWX SC uses drbd luks storage layer stack with allowRemoteVolumeAccess: "true".

Happy to capture journal dumps, drbd-reactorctl status -v, or drbdsetup status -vvv from a live broken state if useful.

[WanzenBug]

Delete the consumer pod. CSI issues NodeUnpublish/NodeUnstage; drbd-reactor demotes the resource; the templated unit chain (drbd-promote@, mount-export@, nfs-ganesha@, advertise-nfs-endpoint@) deactivates.

drbd-reactor should not demote the resource. The resource should stay promoted, regardless of if a consumer pod exists or not.

The other speculation sounds plausible at least. If I find the time, I will set up a test lab using the provided environment details...

[bernardgut]

@WanzenBug naah WAIT. Dont waste your time yet. THis is largely hallucinated by Claude Opus. I was in the middle of a 12h debug session, in the middle of a saturday night migration that turned into a weekend long migration. Basically looking at finding why it didnt work. Give me a min I am writing what I actually found.

[bernardgut]

You're right: the promoter has no link to CSI or pod lifecycle. NodeUnpublishVolume/NodeUnstageVolume against an NFS-fronted PVC only touch the NFS-client mount on the workload node; the gateway-node units (drbd-services@<res>.target and everything under it) stay up. Sorry for the confusion.

What I think I actually hit, retrospectively: drbd-reactor was IGNORING the resource on that node, so it never issued systemctl start drbd-services@.target, and the chain stayed in its last Skipped/dependency state. In my case journalctl -u drbd-reactor showed one of:

• IGNORING resource '...': missing field 'connections' at line N column 5 : the resource was running with autoPlace=1, so the rendered .res file had no peer-connections block.
• IGNORING resource '...': quorum is 'off', but also fencing is 'dont-care' : promoter requires at least one split-brain safety mechanism.

Adding a Diskless replica on a peer and setting DrbdOptions/Resource/quorum=majority cleared it within seconds. The "delete the helper pod" workaround I cited earlier was incidental. I think The reactor reload happened to coincide with the config fix.

Nothing here applies to @utegental's original report. Apologies for the noise. I was desperate and trying to find a solution/something to blame and didn't really take the time to figure it out before posting (which I usually always do). I will edit the post above