Add a `sucoder nodes [PARTITION]` query: show available compute nodes without an interactive login

[ligon]

Motivation

When Lustre is degraded, some savio3 compute nodes hang on I/O while others are fine. Before reserving a node (or targeting one with --node / --local-disk), it would help to see which nodes in a partition are actually free and healthy. Today that means shelling into a login node interactively (PIN+OTP) and running sinfo by hand.

Everything needed to do this non-interactively already exists in sucoder: each remote target carries a gateway and a 24h ControlMaster socket (~/.sucoder/ssh/<gateway>.sock), and the slurm targets carry a partition. A small query command could reuse that warm socket and run sinfo for us.

Proposal

A node-availability query, e.g. sucoder nodes [PARTITION] (this is the --nodes-available query I asked for; a subcommand is probably the more idiomatic typer shape, but either works):

• sucoder -T savio-node nodes -> defaults the partition from the target's slurm.partition (savio3).
• sucoder -T savio-htc nodes -> savio4_htc.
• sucoder -T savio-node nodes savio3_gpu -> optional positional overrides the partition (gpu, bigmem, etc.).

Output: one row per node with state, CPUs Allocated/Idle/Other/Total, and load, plus the drain reasons. Roughly:

sinfo -p <partition> -N -o "%N %6t %.15C %.6O"
sinfo -p <partition> -R          # drained/down nodes + reasons

Implementation sketch (reuse existing machinery)

• Resolve the target (-T) the way other subcommands do; partition = optional positional arg, else remote.slurm.partition.
• Run sinfo on the login node over the existing gateway ControlMaster socket (ControlPath=~/.sucoder/ssh/<gateway>.sock, ControlMaster=no, BatchMode=yes) so it costs no OTP when a tunnel is already warm; fall back to opening a tunnel if the socket is dead. tunnel.SshControl / _control_socket_path() already model this.
• No new config; read partitions from the target's slurm block.

Composes naturally with the existing --node (adopt/target a node) and --local-disk flags: see which nodes are free, then aim at one.

Honest caveat

sinfo reports SLURM availability, not Lustre client health: a node can read idle while its Lustre mount is wedged. So -R (drain reasons, since admins often drain sick nodes) and an anomalous load on an otherwise-idle node are the best proxies the query can surface; it can't promise a node's filesystem is healthy. Worth a line in the output/help so the result isn't over-trusted.

---

Filed by Sue at Ethan's request (Letta Code session), 2026-06-15.

[ligon]

Implemented in 5c99cce (feat(cli): add sucoder nodes [PARTITION]), landing on main — this issue will auto-close on push.

Assessment

The proposal checked out against the code: SshControl + _control_socket_path (tunnel.py), ssh_options(with_fallback=…), SlurmConfig.partition/optional slurm block, and resolve_target all exist as described, and the decisive precedent — squeue run over ln_control.ssh_options(with_fallback=True) in _build_executor — is the exact shape the new query needed.

One nuance from the issue was tightened during implementation: "run sinfo over the gateway socket" is right for BRC specifically because _build_executor pins the login node via ssh <gateway> hostname — i.e. ssh <gateway> already lands on a login node where the SLURM clients live. So nodes runs sinfo directly over the warm gateway mux with no login-node pinning ceremony; only the gateway hop is OTP-gated, and that's the one that's kept warm.

What shipped

• sucoder -T <target> nodes [PARTITION] — partition defaults to the target's slurm.partition, optional positional overrides it.
• sinfo -N (state, CPUs A/I/O/T, load) + sinfo -R (drain reasons) over the warm gateway socket; no OTP when the tunnel is up, falls back to an OTP prompt only when cold.
• sinfo data → stdout, setup chatter + the Lustre-health caveat → stderr (pipes cleanly).
• The honest caveat is surfaced in --help and output: sinfo reports SLURM state, not Lustre health.
• No new config. 13 tests; documented in README.org.

Adversarial review (3 reviewers + 1 verifier)

Found and fixed before commit:

1. Drain output was inverted — sinfo -R always emits a header, so the healthy "0 drained" case printed a bare header and (none reported) only fired when the query failed. Now: header-only ⇒ "(none reported)"; a failed query ⇒ explicit "(could not query drain reasons)".
2. No subprocess timeout — a wedged-but-alive mux could hang nodes forever. Added a wall-clock timeout → synthetic exit-124, matching the timeout-guarded probes in tunnel.py.
3. Partition validation — added a charset/leading-- guard (plus shlex.quote as defense-in-depth) so a partition value can't be mistaken for an sinfo flag or break out of the remote shell.

Deferred (LOW, documented): the gateway-runs-sinfo assumption is correct for BRC and noted in a code comment; it would only need revisiting for a pure-bastion target that lacks the SLURM clients.

— Sue