Replace orchestrator pools with a single takeover RPC

[hermabr]

Minimal rework of the orchestrator-takeover design (supersedes the two-message inventory/claim version). A new run started with FURU_REPLACE_ORCHESTRATOR inherits a live run's Slurm workers instead of re-queueing them.

Design: one RPC, each side does its own local work

The successor prepares its snapshot, venv, and worker config before connecting — that's the same work a cold start needs, so nothing is wasted if the takeover matches nothing. It then sends one /takeover request: its coordinates, one offer per Slurm backend. The old coordinator does everything local to it:

1. matches offers to its pools by fingerprint,
2. stops scaling and rewrites the matched pools' endpoint files to the offered coordinates,
3. signals its running workers with scancel --signal=USR1,
4. answers with the adopted pools' job ids and shuts itself down.

The successor never touches another run's files or Slurm jobs. Besides being smaller (two wire models instead of six, no mid-handshake state where the successor holds half-finished writes into someone else's directory, no separate "drop before claim" path), this is the division of labor a multi-cluster future needs: only the side that owns a cluster has filesystem and scancel access to it, so "redirect yourself to me" is the message shape that survives, even though nothing multi-cluster is built here.

Worker-side machinery is unchanged from the previous design: sbatch scripts resolve the coordinator URL, token, project, and config through a per-pool endpoint file at runtime, and a disconnected worker re-reads the file and re-execs when the generation grew. The filesystem live-run registry remains as a dumb discovery pointer ({url, token}).

Failure behavior

• A takeover that matches nothing changes nothing: the old run only surrenders and stops after adopting at least one pool, and the destructive step is gated on a fully validated request.
• A dropped or malformed connection before the request surrenders nothing.
• A worker that misses the USR1 signal still redirects at its next touch of the dead old socket.

Note: tests/test_furu_locking_contention.py::test_lock_is_taken_over_mid_create is flaky on slow filesystems (0.5s spawn deadline) — it fails identically on unmodified main on my machine and is unrelated.

🤖 Generated with Claude Code