fix(democratic-csi): unique NVMe host NQN per node

[david-igou]

Closes #404.

Problem

All democratic-csi NVMe-oF node plugins connect to TrueNAS with the same host NQN uuid:941e4f03-..., baked into the democratic-csi container image. The DaemonSet runs one image on every node and we don't mount the host's /etc/nvme (node.driver.nvmeDirMountEnabled defaults false), so every node presents to TrueNAS as the same NVMe initiator — no per-host isolation at the fabric layer, and a data-integrity risk if two nodes ever attach the same namespace. Confirmed identical on ocp/hpg5/p330.

Fix

The chart's node template has no initContainers hook, so this is a targeted kustomize patches entry (patches/nvmeof-unique-hostnqn.yaml) applied by name regex to the three NVMe-oF node DaemonSets (ssd/fast/cold):

• An initContainer derives a deterministic host NQN + host ID from the node's /etc/machine-id (32 hex = a 128-bit id, unique per node and stable across reboots), formats it into UUID shape, and writes hostnqn/hostid into an emptyDir mounted over /etc/nvme in the csi-driver container.
• Deterministic (not nvme gen-hostnqn random) so the NQN is stable — reconnects and any future TrueNAS host ACL keep matching.
• Reuses the node-plugin image (already present on every node; no extra pull) just for sh+coreutils.

Validation

• kustomize build --enable-helm renders the initContainer + emptyDir + csi-driver mount on exactly the 3 nvmeof node DaemonSets; the other 6 (iscsi/nfs) are untouched.
• yamllint clean.
• UUID derivation verified against real machine-ids: valid UUID shape, unique per node (ocp ce8235e7-..., hpg5 de69ca75-...).

Rollout notes

• Applying restarts the 3 nvmeof node DaemonSets. Already-attached volumes keep their existing (old-NQN) kernel connections (ctrl-loss-tmo=-1); new attaches and post-reboot reconnects use the unique NQN, so it converges as volumes cycle / nodes drain. nvme disconnect on teardown is unaffected (keyed on device/subsystem, not host NQN).
• discovery.conf is intentionally not recreated in the emptyDir — the freenas-api-nvmeof driver connects to explicit transports from its config and doesn't use nvme discovery.

Follow-ups (not in this PR)

• iSCSI initiator-name (/etc/iscsi/initiatorname.iscsi) duplication was not conclusively verified (host file absent); tracked as the NVMe-oF: democratic-csi node plugins share one host NQN across all nodes #404 iSCSI check.
• Host /etc/nvme/hostnqn duplication (serves only each node's local boot disk) remains as low-priority cosmetic cleanup.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ABCDdxkj3rbL6jF1QA5EwM

[david-igou]

Upstream research: this defect is tracked as democratic-csi#451, and this patch is the right fix to keep

Checked whether upstream democratic-csi already solves or tracks the baked-in hostnqn, to decide if we should prefer a native mechanism over this patch. Findings (source-grounded):

Tracked upstream, open: democratic-csi/democratic-csi#451 — "NVMEoF driver uses NQN configured during container build stage" (open since 2025-02-04). It's the only hostnqn-related issue/PR in either repo and describes our exact defect.

Root cause confirmed verbatim in source: the image Dockerfile (lines 121-122, on both next and master) literally bakes

echo 'nqn.2014-08.org.nvmexpress:uuid:941e4f03-2cd6-435e-86df-731b1c573d86' > /etc/nvme/hostnqn

— the exact NQN we observe on every node.

Upstream's attempted fix is dead code: the maintainer added generate-if-missing logic to next (bin/democratic-csi: if (!fs.existsSync(...)) nvme gen-hostnqn), but because the Dockerfile still bakes the file, existsSync is always true and it never runs under the default config. So even the latest next image ships duplicate NQNs. There is no chart release past 0.15.1.

No simpler native option exists:

• nvmeDirMountEnabled: true only surfaces the host's /etc/nvme, and RHCOS hosts share a duplicate hostnqn too (Red Hat KB 7073579) — so it fixes nothing without an added per-node MachineConfig + reboots.
• democratic-csi's nvme connect never passes --hostnqn/--hostid (src/utils/nvmeof.js connectByNQNTransport), so /etc/nvme/hostnqn is authoritative — exactly what this patch's emptyDir controls.

Why this patch is actually better than upstream: deriving deterministically from /etc/machine-id gives a stable NQN across pod reschedules; upstream's random-on-first-start would remint the NQN on every reschedule. The patch also correctly writes both hostid and hostnqn, uses a valid NQN format, and omits discovery.conf (democratic-csi never uses nvme discovery).

Removal condition (added to the patch header comment): drop this patch only once #451's Dockerfile stops baking hostnqn/hostid and a persistent per-node mount lands upstream.