DO-NOT-MERGE: RFC: NVMe-oF orchestrator coexistence — nvme-discoverd, ownership registry, and exclusion list

[martin-belanger]

This PR is for review only and will not be merged. It contains four RFC documents proposing a coordinated set of features for NVMe-oF orchestrator coexistence in nvme-cli 3.0. Please use inline comments to provide feedback on specific sections.

Tip: Markdown files are shown as raw text by default in the diff view. Click the "Display the rich diff" button (the document icon at the top-right of each file) to render them as formatted text — much easier to read.

---

Background

As NVMe-oF deployments grow, hosts increasingly run multiple tools that manage NVMe-oF connections: nvme-stas, dracut/initramfs scripts, and soon nvme-discoverd. Without coordination, these orchestrators can conflict — one may disconnect a controller that another is actively managing, or connect to a controller that another has deliberately excluded.

This RFC set proposes two libnvme building blocks to solve the coexistence problem (ownership registry and exclusion list), and a new daemon orchestrator (nvme-discoverd) that puts them to use.

---

The four RFCs

rfc-nvme-orchestrator-coexistence.md — Start here

The top-level document. Frames the two distinct conflict scenarios (accidental disconnect, accidental connect), introduces the two prevention mechanisms (ownership registry and exclusion list), defines a three-tier orchestrator hierarchy (raw commands → manual orchestrators → daemon orchestrators), and shows how nvme-discoverd and nvme-stas naturally partition work without requiring IPC coupling between them.

rfc-nvme-registry.md — Ownership registry

A lightweight, cooperative registry under /run/nvme/registry/ that lets orchestrators declare ownership of connected controllers. nvme disconnect-all consults the registry before acting so it never disconnects a controller managed by a running daemon. The registry is a libnvme building block: a C API (libnvmf_registry_*), a Python binding, and a nvme registry CLI command family. An implementation PR already exists: #3425.

rfc-nvme-exclusion.md — Exclusion list

A human-administered exclusion list at /etc/nvme/exclusions/ that prevents orchestrators from auto-connecting to controllers the administrator wants excluded. Its design center is auto-discovered controllers (mDNS, CDC DLP) where there is no configuration entry to remove. Managed via nvme exclusion CRUD commands. Enforcement is cooperative — each orchestrator reads the list and skips matching controllers; libnvme does not enforce it.

rfc-nvme-discoverd.md — nvme-discoverd

A new daemon proposed for nvme-cli 3.0. It manages NVMe-oF connections for statically configured controllers, NBFT boot controllers, FC-discovered controllers, and (in a future release) mDNS-discovered controllers. Key design choices:

• Connect-only by design — never issues disconnects; eliminates the entire connect/disconnect ordering complexity (CtrlTerminator problem). TP8010 fabric zoning is out of scope; that belongs to nvme-stas.
• One systemd transient unit per controller — nvme-discoverd never calls libnvmf_connect() directly; blocking /dev/nvme-fabrics writes happen in child processes managed by systemd. Achieves Daniel's no-threading goal.
• Registers ownership via --owner discoverd on each generated unit; NBFT reconnects use --owner nbft to preserve the lifetime invariant set at boot.
• Respects the exclusion list before each connect and reconnect; NBFT controllers bypass the check unconditionally.
• Coexists with nvme-stas through natural discovery partitioning: nvme-stas owns mDNS/TCP, nvme-discoverd owns NBFT/FC/RDMA and manual config. The registry-check rule in nvme-stas eliminates bounce loops without any IPC coupling.

---

What feedback is sought

• Architecture: Does the three-tier orchestrator model match how you see these tools being used in practice?
• Ownership registry: Any concerns with the directory layout, API shape, or cooperative-only enforcement model?
• Exclusion list: File format, use cases, nvme exclusion command design?
• nvme-discoverd: Transient unit design, state file layout, startup reconciliation logic, retry policy, NBFT handling?
• Coexistence rules: The owner=nbft lifetime invariant, NBFT exclusion bypass, unowned-connections-are-fair-game semantics — do these cover the cases you care about?

---

Related

• Implementation PR for the ownership registry: libnvmf: NVMe Controller Ownership Registry (V2) #3425 (branch registry-v2)
• Previous registry discussion: Prevent NVMe-oF boot devices from being removed by disconnect-all #2913
• Feature tracking issue: libnvme/nvme: add NVMe controller ownership registry #3433

[MichaelRabek]

I don't see having our own implementation of MD5 as necessary since we already depend on openssl for fabrics support. Also, why not use SHA1 for example?

[martin-belanger]

Two things.

First, nvme-cli does not unconditionally depend on openssl — it's an optional build. openssl is a meson option (auto, and can be explicitly disabled), and fabrics builds fine without it; you only lose TLS and DH-HMAC-CHAP authentication, not the transport. discoverd has to work in those builds too, so I deliberately did not tie a core naming operation to openssl being present.

Second — and more to the point — the hash here is not cryptographic. It only turns a long, variable-length identifier into a short, stable, filesystem- and systemd-safe token for a unit/file name. There's no adversary, no security property, and no collision-resistance requirement in the cryptographic sense — I just need something fast, reliable, and dependency-free. So MD5 vs SHA1 vs anything else is immaterial, and I'm not attached to MD5 at all.

Given that, the cleanest answer is probably neither MD5 nor openssl: nvme-cli already ships a self-contained non-cryptographic hash (util/crc32, plus util/base64), so the token can be derived from that — no new dependency, and no hand-rolled crypto. Happy to use whichever in-tree option you prefer, as long as it doesn't make the unit naming depend on openssl.

[dwsuse]

Same here, as long the length of the created hash value is relatively small :)

Pure md5 creates 16 byte hashes which seems to be the shortest hashing algorithm. So we might even want to cut off some of the bytes.

Or we just try it with crc32 as this gives us 4 bytes and see how this goes.

This is relatively easy to change later.

[martin-belanger]

Agreed on dropping MD5 for a small in-tree non-cryptographic hash; the only refinement is width. Pure crc32 is 32 bits → 8 hex chars, but at our lab scalability reference (~200 concurrently-connected controllers) the birthday-bound collision probability is ~1 in 215,000 — closer than I'd like for something that names a system unit. Widening to 48 bits → 12 hex chars costs only 4 more characters and drops that to ~1 in 14 billion at 200 controllers. So I'll use a 48-bit FNV-1a token (truncated) rather than 32-bit crc32 — same idea (small in-tree hash, no crypto dependency, easy to change later), just a touch wider. I'll also add an at-creation collision check: discoverd already holds the TID→unit-name map in memory, so any collision is detected and logged rather than silently leaving a controller unmanaged. I'll update §3.2.1 from MD5/32-hex to a 48-bit/12-hex hash with that guard.

[MichaelRabek]

This is just personal preference, but I generally don't like seeing things on my system named with cryptic names like UUIDs and/or hashes. I would like to get a clue what subsystems am I connected to immediately when running systemctl status.

How about we use nvme-discoverd-<NQN>-<hash>.service unit name format, where we'd only use a couple of digits from the hash in a similar way to git log --oneline?

[igaw]

FWIW, nqn are potentially very long. I've played with a simple mapping which included the subsysnqn and was not too happy with the result either because the service name was well above 120 chars.

[martin-belanger]

Building on Daniel's point — putting the NQN in the unit name has two problems:

• Length. NQNs are spec-limited to 223 bytes, so nvme-discoverd-<NQN>-<hash>.service can run well past the 255-byte filename limit, and is unwieldy long before that (Daniel's >120-char experience).
• Escaping. NQNs are full of characters systemd must escape in a unit name — : and . (e.g. nqn.2014-08.org.nvmexpress:uuid:…) become \x3a etc. So the "readable" NQN isn't actually readable once it's a unit name; it ends up both long and mangled, which defeats the purpose.

But I do want to solve the real need — seeing what you're connected to from systemctl status. The clean way is to keep the unit name short and stable (hash-based) and put the human-readable identity in the unit's Description=, which is exactly what systemctl status prints. You'd get something like:

nvme-discoverd-<short-hash>.service - Discovery: nqn.2014-08.org.nvmexpress.discovery @ 192.168.1.3:8009

i.e. the clue you want, in the place you'd look for it, without paying the length/escaping cost in the unit name. Would that address the concern?

[dwsuse]

Good point, the escaping of the nqn was also not nice. I think if we knew the ctrl name before hand we would just name it nvme-discoverd-nvme$x and the rest in the description. In this sense the hashing of the tuple is generating a stable id for the service which makes sense to me. As long the hash is relative short.

[martin-belanger]

Agreed on both counts. The hash is precisely there to get a stable id before the device name exists (we can't name the unit nvme-discoverd-nvmeX because X isn't assigned until after the connect completes). And yes, short: I'll use a 48-bit hash rendered as 12 hex characters (nvme-discoverd-<12hex>.service), which stays compact while keeping collisions negligible even at a few hundred controllers (width rationale in my reply on the hash-algorithm thread).

[MichaelRabek]

I don't think nvme-stas should be reading /etc/nvme/discoverd.conf to get such information. Configuration files of other programs should not be of concern to nvme-stas. This would create a not so pretty dependency in nvme-stas on the format of the configuration file.

Wouldn't it be better if nvme-stas got this information from nvme-discoverd through some IPC primitive?

[martin-belanger]

Agreed. Reading /etc/nvme/discoverd.conf from nvme-stas was a shortcut chosen purely for implementation simplicity, and you've put your finger on its real flaw: it couples nvme-stas to discoverd's config-file format, so any change to that format silently breaks the check — no error, just a mismatch that no longer gets detected. That's brittle.

The robust answer is the IPC one you suggest. nvme-discoverd already exposes a varlink interface, so nvme-stas should query the state it cares about (whether discoverd has mDNS/zeroconf enabled) over that socket rather than parsing a file it doesn't own. It costs more code on both sides — a varlink method in discoverd and a client call in stas — but it's the correct boundary, and it doesn't break when either side changes its config syntax. I'll update the RFC to specify the IPC approach and drop the config-file read.

[dwsuse]

Not sure about the first point needs to be in the documentation.
IMO,, the main points are

• Static configuration, no filtering, exclusion
• manages global resource without 3rd party to hook into

[martin-belanger]

Agreed — I'll reframe around what discoverd is rather than editorializing about the udev rules. The positioning you list is the right spine:

• it is a static-configuration connector — no DLP-delta reconciliation, no per-controller filtering/exclusion logic of its own (exclusion is a separate, system-wide mechanism it merely consults);
• it manages a global resource (the host's fabric connections) directly, without requiring a third party to register callbacks or hook into it.

I'll keep retry-on-failure and the planned mDNS/TP8009 support as concrete forward-looking motivators, and drop the "udev rules are complex" point as a stated motivation.

[dwsuse]

I would not mention the downstream packaging plans.

[martin-belanger]

Agreed. This is an upstream RFC; the SUSE sl16.2 release-cycle reference doesn't belong here. I'll remove it and keep the targeting generic (nvme-cli 3.0).

[dwsuse]

nvme-discovered does the discovery and creates service files which handle the connection. Thus I am not sure what the 'connect-only by design'.

[martin-belanger]

Fair — the phrase is doing too much work. What I mean is narrower than it reads: discoverd never disconnects a live controller in steady state. It has no stateful DLPE-delta path that decides "this controller dropped out of the log page, tear it down" — that's TP8010 reconciliation, which belongs to nvme-stas. Disconnects happen only as a side effect of a systemd unit stop (shutdown / manual stop), never as a discovery-driven decision.

So "connect-only" is about the disconnect policy, not about discoverd being passive — you're right that it discovers and drives connections. I'll reword it to something like "non-disruptive: discoverd never tears down a live controller in response to discovery state" so the meaning is unambiguous.

[dwsuse]

Ah nice! Let's see how this goes. A quick documentation on StartTransientUnit gave me also an AI answer which claimed these service types can't be controlled with 'systemctl start/stop'. Which kind makes sense. Anyway, let's give this a try and see how it goes.

[martin-belanger]

The AI answer is wrong, or at least imprecise. A transient unit created via StartTransientUnit is a normal unit in systemd's table once created — systemctl stop nvme-discoverd-<hash>.service works and is exactly what runs the ExecStop= disconnect, both at shutdown (via the Before=/After= ordering) and on demand. RemainAfterExit=yes is what makes stop meaningful for a Type=oneshot unit: without it the unit would already be inactive after ExecStart= exits and there'd be nothing to stop. What you can't do is systemctl start it again after it's been garbage-collected (it has no on-disk unit file) — that's discoverd's job via StartTransient/RestartUnit. So: stop yes, restart-from-scratch via systemctl no. Agreed it's worth a real try — I'll note this distinction in §3.2.

[dwsuse]

This will not work. When the service file is created with nvme connect, we don't know the device name yet. Thus service file can't have a nvme disconnect -d /dev/nvme$x in it. This was the reason why I made the nvme connect a deamon.

I think what we could/should do first, is to extend nvme disconnect to accept the exact same arguments nvme connect accepts which then are used to lookup the controller to shutdown. This would avoid the daemonizing of the nvme connect

We can still implement a nvme-connectd service if needed. But if we keep it simple with nvme-cli, I would like to try the nvme disconnect accepting all the connect args.

[dwsuse]

BTW, the nvme-connectd might make even sense for retry/error logic. nvme connect does not have any retry loop at this point and we don't handle transient errors. For my plans to make FC writes to '/dev/nvme-fabrics' blocking it is a necessity to handle transient errors.

OR another idea is to use the registry to store some ID in there which we use to identify the device to disconnect. Not so sure about this though as it only solves half the problem.

[martin-belanger]

I think this is already solved in the RFC — it's the point of §3.2.4 ("Capturing the device name") and §3.2.5, which come right after this line, so it's easy to have read past. The unit does not bake a device name into ExecStop=. Instead:

• ExecStart= carries a new nvme connect --devid-file=%t/nvme/discoverd/units/%N.devid option. On success nvme connect writes the kernel-assigned nvmeX name into that file. For an already existing controller (idempotent) the name comes from libnvme's sysfs scan, so the file is populated in that case too.

• ExecStop= reads it back and disconnects by that name:

  ExecStop=-/bin/sh -c 'DEV=$(cat %t/nvme/discoverd/units/%N.devid 2>/dev/null); \
      [ -n "$DEV" ] && nvme disconnect -d $DEV'

The %t (runtime dir) and %N (unit name) specifiers make the same literal path valid in ExecStart=, ExecStartPost=, ExecStop= and ExecStopPost=, so the device name is handed from connect to disconnect through a per-unit file with no value embedded at unit-creation time and no daemon in the loop. RuntimeDirectoryPreserve=yes keeps the file alive across a discoverd restart (§3.2.2, §3.8).

So the device name is known by the time ExecStop= runs — it's captured at connect time and persisted per-unit. This is exactly the problem that pushed you toward daemonizing nvme connect, and the --devid-file handoff avoids needing the daemon for this purpose.

On your alternative — teaching nvme disconnect to take the full nvme connect argument set and re-derive the controller by matching the TID in sysfs — I think that's a genuinely good addition on its own merits (it's the natural symmetric CLI, useful to humans and to any caller that didn't capture the device name). I'd be happy to see it land. For discoverd specifically I'd still prefer --devid-file, because it disconnects the exact device this unit connected rather than re-resolving a TID→device match at stop time (which has to handle "what if two controllers share a TID view in sysfs" and "what if the match is ambiguous"). The two aren't mutually exclusive: --devid-file is the precise path for the unit; arg-based nvme disconnect is a nice general capability. If you'd rather not add --devid-file at all, arg-based disconnect is a workable fallback for discoverd — it just trades a captured-name lookup for a re-derivation.

[martin-belanger]

Regarding nvme-connectd:

Two separate things:

• Retry / transient-error handling. For discoverd's purposes the retry loop lives in the orchestrator, not in nvme connect: a failed connect leaves the unit failed, and discoverd's reconnect logic (RestartUnit / re-StartTransient on the next trigger, §3.2.6, §3.5) drives the retry. So discoverd doesn't need nvme connect to grow a retry loop. That said, your FC-blocking-write work is a different motivation — making /dev/nvme-fabrics writes block and handling transient FC errors at the connect layer is valuable independent of discoverd, and nvme-connectd may well be the right home for it. I'm not arguing against nvme-connectd in general; I'm arguing discoverd doesn't require it just to know the device name.

• Registry-stored ID. The ownership registry is already keyed by the nvmeX device name (§3.6, same convention), so the "ID to identify the device to disconnect" would be the device name we're already capturing in the .devid file — storing it a second time in the registry would be redundant, and as you note it only solves half the problem (you still need to capture it at connect time, which is the --devid-file step). So I'd keep the device-name handoff in the per-unit .devid file rather than route it through the registry.

[dwsuse]

When testing with larger setup (>= 200 namespaces, 8 paths, 80 CPUs) , 'time nvme disconnect' gives values in the range of 30-40 seconds without any error. So this could easily run way longer. So we really need to figure out how to handle this scenario here.

Obviously, we should also look into the making the disconnect path faster. I think the main reason why this is so slow is the extremly heavy use of rcu_sync in the shutdown path. IIRC there are 5 rcu_sync in this path per namespace...

[martin-belanger]

Good data — and it directly invalidates the TimeoutStopSec=10 I put in §3.2.3/§3.4. If a healthy disconnect can take 30–40 s (and more on larger configs), a 10 s stop bound would SIGTERM the disconnect mid-flight and leave the controller half-torn-down — worse than no bound. I'll fix this:

• Raise the default substantially (well above your observed worst case) rather than the 1–3 s assumption the draft was written against, and
• make it configurable (a discoverd.conf knob, or just inherit a generous systemd default) so large deployments can tune it, instead of hard-coding a single number.

The point of the bound was only to avoid systemd's 90 s default hanging shutdown on an unresponsive target — but a slow-but-progressing disconnect is the common case at scale, so the bound has to sit above realistic completion time, not below it. I'll rework §3.2.3 to say that and drop the "1–3 s typical" claim.

The kernel-side rcu_sync cost in the disconnect path is a separate (and very worthwhile) optimization target, but it's orthogonal to the RFC — discoverd has to tolerate whatever the disconnect latency is, and the fix here is the timeout policy.