refactor(runtime): split operator runtime into per-language crates

[phil-opp]

Splits the monolithic dora-runtime into a language-neutral SDK plus per-language backend crates, and table-drives the daemon's runtime spawn logic. Behavior-preserving — no descriptor/YAML changes.

• dora-runtime-api — the SDK: operator event loop, node harness, and the OperatorRunner backend trait + main(runner) entry.
• dora-runtime-shared-lib — libloading/C-ABI backend, shipped in the dora CLI (dora runtime).
• dora-runtime-python — PyO3 backend, the only crate linking pyo3, shipped in the wheel (dora.start_runtime()).

The python cargo feature and all #[cfg(feature = "python")] dispatch are gone; adding a language is now one crate implementing OperatorRunner. The daemon's python/shared-library selection moves to spawn/runtime_registry.rs, keyed on the new OperatorSource::runtime_name() helper, reproducing the existing launch commands verbatim (incl. the #1797/#1805 fixes). The registry is the seam where a follow-up can add third-party runtimes via an explicit runtimes: map.

Verified: full cargo test --all, clippy --all -D warnings, fmt --check, cargo check --examples, and an end-to-end shared-library operator dataflow (daemon → runtime → dlopen'd operator → sink, all green).

[trunk-io]

😎 Merged successfully - details.

[phil-opp]

🤖 Automated review by Claude — this review is fully automated and has not been verified by a human.

I reviewed this refactor against the pre-split behavior, focusing on the daemon spawn path and both backends. It looks behavior-preserving:

• runtime_registry::runtime_command reproduces the original Python-vs-native classification exactly (runtime_name() == RUNTIME_PYTHON is equivalent to the old matches!(.., OperatorSource::Python{..})), including the mixed-language bail!, and native_runtime_command preserves the daemon: spawn runtime via current_exe() instead of which("dora") #1797/Nightly regression since 2026-05-13 #1805 current-exe handling verbatim.
• The SharedLibRunner / PythonRunner fallback arms keep the old behavior where a mis-routed operator surfaces as an init failure via the closed oneshot.
• OperatorSource::resolve() / runtime_name() are covered by the new serde round-trip tests.

No correctness or logic bugs found.

One thing worth confirming (low confidence, non-blocking): the telemetry feature on dora-runtime-python / dora-runtime-shared-lib still pulls in tracing-opentelemetry, but the event loop now lives in dora-runtime-api::main, which only configures tracing via dora_tracing::TracingBuilder behind the plain tracing feature. None of the split crates reference tracing_opentelemetry, so the telemetry feature may no longer wire OTLP span export the way the monolithic crate did. It's non-default, so normal builds are unaffected — just worth a check that OTLP tracing isn't silently dropped for telemetry users.

---

Generated by Claude Code

[phil-opp]

🤖 Automated review by Claude — this is a fully automated review with no human in the loop. Treat it as a suggestion to verify, not as authority.

I found one issue worth a look. This PR is billed as behavior-preserving, but I think it narrows behavior for shared-library operators hosted under a Python-embedded daemon (i.e. a pip-installed dora, where the daemon's current_exe is the Python interpreter).

Before the split, the unified dora-runtime handled OperatorSource::SharedLibrary unconditionally in run_operator (binaries/runtime/src/operator/mod.rs) — the python cargo feature only added the Python arm. So the runtime shipped in the wheel (built with python) could host both native and Python operators.

The daemon's routing is reproduced verbatim: native_runtime_command (binaries/daemon/src/spawn/runtime_registry.rs) still sends non-Python operators to python -uc "import dora; dora.start_runtime()" when current_exe ends in python/python3 — identical to the old spawner.rs path. But in the new wheel, dora.start_runtime() now binds to dora_runtime_python::main() → PythonRunner, which has no libloading dependency and takes the OperatorSource::SharedLibrary(_) arm: it logs an error, returns Ok(()), and drops init_done — surfacing to the operator as an init failure.

So for a pip-installed dora (daemon exe = the Python interpreter) running a runtime node with a C/C++/Rust shared-library operator, the launch command is reproduced exactly but the launched backend can no longer host the operator. This path isn't caught by CI: the C++/cmake shared-library examples run through the compiled dora binary (current_exe == "dora" → dora runtime → the shared-lib backend) and never hit the python-spawned route.

If that configuration is still meant to be supported, the native_runtime_command python branch should launch the shared-lib backend rather than dora.start_runtime() (or PythonRunner should be able to delegate shared-library operators). If it's intentionally unsupported now, a note in the PR description would help, since the change is described as behavior-preserving.

---

Generated by Claude Code

[phil-opp]

🤖 Automated review by Claude — fully automated, no human in the loop; it may contain mistakes.

Went through the split carefully — the mechanical moves look faithful (the runtime_name()-based classification in runtime_registry is equivalent to the old matches!(Python{..}) split, the session.rs kind_tag literals are unchanged so the build hash is stable, and the renamed runner files are pure moves).

One correction/refinement on the shared-library-under-Python regression noted in the automated comment above: it doesn't actually hit a normal pip install + dora up/dora start. In that mode the pip dora console binary has file_name == "dora", so native_runtime_command routes to dora runtime → SharedLibRunner and native operators still work. The regression is specifically when the daemon itself runs as an embedded Python process (current_exe ends in python/python3, e.g. python -c "import dora; dora.start_daemon()"): there native_runtime_command sends native operators to python -uc "import dora; dora.start_runtime()", which now binds to PythonRunner and can no longer dlopen them. The old wheel runtime (built with the python feature) handled SharedLibrary unconditionally, so this is a genuine narrowing of behavior despite the "behavior-preserving" framing. Worth either wiring that branch to the shared-lib backend or explicitly calling it out as unsupported.

Also flagging that the PR currently shows a merge conflict against main.

---

Generated by Claude Code

[phil-opp]

🤖 Automated review by Claude — this is a fully automated review with no human in the loop; treat findings as suggestions to verify, not as authority.

I went through the split again focusing on the release wiring rather than the spawn path (earlier comments already cover that). One thing looks like it'll break at release time and isn't caught by cargo test / cargo check --examples:

The dora-runtime crate rename isn't reflected in the release workflows. This PR renames the published dora-runtime crate to dora-runtime-python and adds dora-runtime-api and dora-runtime-shared-lib (workspace Cargo.toml), so there is no longer a crate named dora-runtime in the workspace. But both publish workflows still name the old crate, and neither file is in this PR's diff:

• .github/workflows/cargo-release.yml → publish_if_not_exists dora-runtime
• .github/workflows/release.yml → dora-runtime in the CRATES=(...) list

On the next crates.io release I'd expect two failures:

1. Publishing dora-runtime fails / errors out because that package no longer exists in the workspace (cargo metadata finds no such package).
2. The new crates never get published, and since the published dora-cli now depends on dora-runtime-shared-lib (→ dora-runtime-api), cargo publish -p dora-cli will fail unless dora-runtime-api and dora-runtime-shared-lib are published before it.

Replacing the dora-runtime entry with dora-runtime-api + dora-runtime-shared-lib, ordered before dora-cli (and dora-runtime-python too, if it's still meant to be on crates.io — note it now links pyo3/abi3 and is shipped in the wheel, so it may want publish = false instead), in both workflow files should cover it.

For context, the embedded-Python shared-library routing and the telemetry/OTLP questions from the earlier comments still stand, and the branch currently shows a merge conflict against main.

---

Generated by Claude Code

[phil-opp]

🤖 Automated review by Claude — fully automated, no human in the loop; it may contain mistakes.

Separate from the points raised earlier, one packaging change looks like it will break local --all builds for a subset of contributors:

dora-runtime-python links PyO3 unconditionally but is now a default workspace member with no matching --exclude. binaries/runtime-python/Cargo.toml declares pyo3 = { workspace = true, features = ["eyre", "abi3-py311"] } with no feature gate, its build.rs calls pyo3_build_config::use_pyo3_cfgs() unconditionally, and the crate is added to the root Cargo.toml members. As the crate's own comments note, that build-config probe fails on interpreters below the abi3-py311 floor (e.g. Python 3.10 on ubuntu-22.04) — which is exactly why the old monolithic dora-runtime kept all of PyO3 behind the default-off python feature: to stay out of a plain cargo build/check/test --all.

This PR drops that gating for the Python backend, but the documented --all commands in CLAUDE.md/AGENTS.md still exclude only the three pre-existing -python cdylib crates (they were updated only to rename the crate rows), and .github/workflows/ci.yml isn't touched. Net effect: on any machine whose default interpreter is < 3.11, cargo check/build/clippy/test --all now fails inside dora-runtime-python's build script where it previously succeeded; and since the crate isn't extension-module, cargo test --all also needs libpython at link. CI on ubuntu-latest (Python 3.12) still passes, so this mostly bites contributors locally.

Adding --exclude dora-runtime-python to the documented --all commands (and CI, if it runs --all), or restoring a feature gate the way dora-runtime had it, would keep a default --all build from requiring a Python ≥ 3.11 toolchain.

---

Generated by Claude Code

[phil-opp]

🤖 Fully automated review by Claude — no human in the loop.

Two supplementary points, separate from the (still-valid) telemetry/OTLP, embedded-Python routing, release-workflow, and PyO3 --all-gating threads already on this PR. These are about the added tests and the new public API rather than the spawn path:

The new run_operator tests don't cover the arms the split actually introduces. Both binaries/runtime-python/src/lib.rs and binaries/runtime-shared-lib/src/lib.rs add a wasm_source_returns_descriptive_error test, but that only exercises the OperatorSource::Wasm(_) arm — which is unchanged behavior lifted verbatim from the old operator/mod.rs. The arms this PR newly adds are the cross-language ones: PythonRunner rejecting SharedLibrary(_), and SharedLibRunner rejecting Python(_). Those are precisely the paths that matter for the split's correctness (and for the embedded-Python-daemon case discussed above, where a native operator now reaches PythonRunner and takes the SharedLibrary arm). A test that asserts each backend returns a descriptive error and leaves init_done unsignalled for the specific mismatched source kind that can reach it would lock the #2595-avoidance contract where it's load-bearing — the WASM arm doesn't do that, since WASM can never be mis-routed to a specific backend the way Python↔shared-library can.

OperatorSource::resolve() / ResolvedOperatorSource look unused outside their own tests. runtime_name() is what the daemon and session.rs actually call; resolve() and the ResolvedOperatorSource struct (libraries/message/src/descriptor.rs) appear only in the new descriptor.rs unit tests. If they're staging for the future runtimes: resolver, it may be worth deferring them to the PR that consumes them — they add net-new public API to dora-message (with semver weight on a 1.0.0-rc crate) that nothing calls yet.

The branch also still shows a merge conflict against main.

---

Generated by Claude Code

[phil-opp]

🤖 Fully automated review by Claude — this review was generated end-to-end by an automated agent with no human vetting. Treat it accordingly.

Re-reviewed the rewritten commit against the earlier threads — the five prior points are addressed: shared-library/WASM operators are now delegated from PythonRunner to SharedLibRunner (so the embedded-Python-daemon native path keeps working), both release workflows list the three new crates ordered before dora-cli, the new backend tests cover the cross-language arms with the init_done-unsignalled contract, and descriptor.rs adds only runtime_name() + constants (no unused resolve()). Spawn routing moved verbatim, RunnerGuard preserves the .so-unload-after-join ordering (#2742), and session.rs's kind_tag literals are unchanged so the build hash stays stable.

One residual from the --all gating point: --exclude dora-runtime-python was added only to the cargo test --all commands. dora-runtime-python links pyo3 unconditionally (its build.rs runs pyo3_build_config::use_pyo3_cfgs() with a non-optional pyo3-build-config build-dep), so cargo build --all / cargo check --all / cargo clippy --all still compile its build script and, per that build.rs's own comment, fail on interpreters below the abi3-py311 floor. Those --all commands in CLAUDE.md/CONTRIBUTING.md (and CI's clippy step) weren't given the exclude, so a contributor on Python < 3.11 now hits a failure there that the old feature-gated dora-runtime avoided. CI is unaffected (Python 3.12). Worth either excluding it from the documented build/check/clippy --all commands too, or restoring a probe gate.

(Also still shows a merge conflict against main.)

---

Generated by Claude Code