Investigate faster XDK/plugin launcher builds without sacrificing process isolation or daemon safety

[lagergren]

Summary

The launcher-runtime hardening work removed daemon-global javatools loading, introduced explicit runtime resolution, and moved direct mode onto build-scoped isolated runtime handling. That work is intentionally about correctness and stability first.

The next question is whether we can make self-hosting XDK builds faster, especially for repeated compile/test launcher tasks, without reintroducing the original classloader and daemon-state brittleness.

This issue tracks that second phase.

Why This Matters

Today we have two important truths:

1. DIRECT mode can reduce startup overhead and is attractive for self-hosting scenarios.
2. DIRECT mode still runs inside the Gradle daemon process, so catastrophic in-process failures remain a real stability risk.

We do not want to go back to:

• daemon-global classloader mutation
• pseudo-mutating loader tasks
• runtime state that depends on what happened to run first in the daemon

But we do want to understand whether we can reuse more runtime/compiler state safely and make the XDK build itself faster.

Problem Statement

Investigate whether repeated XTC compile/test/run work can be accelerated by reusing a runtime or worker process keyed by resolved runtime fingerprint, while preserving:

• correctness across different XDK builds and branches
• stability across Gradle daemon reuse
• support for mixed execution modes in the same build
• clear invalidation when runtime contents or plugin/runtime code changes
• the ability to use real javatools APIs programmatically where appropriate

Non-Goals

This issue is not about:

• reviving daemon-global classpath mutation
• making root aggregate builds default to current in-daemon DIRECT mode
• reducing the plugin to string arrays and main() calls everywhere

Candidate Directions

1. Reused external worker/daemon keyed by runtime fingerprint

Potential shape:

• plugin resolves an explicit launcher runtime descriptor
• a build-scoped owner maps runtime fingerprint -> reusable external process
• compile/test requests are sent to that process
• the process can hold warm JVM and possibly future compiler/type-checker state
• crashes stay outside the Gradle daemon

This is currently the most promising direction because it keeps process isolation while still offering reuse.

2. Gradle Worker API with stronger isolation and reuse

Investigate whether Worker API isolation can provide enough reuse and lifecycle control for launcher-heavy workloads, or whether a plugin-managed external process is still the cleaner model.

3. More targeted use of current in-process direct mode

This is lower priority as a default architecture, but we should still quantify where current direct mode helps and where it remains too risky.

Required Constraints

Any design pursued here must preserve:

• no daemon-global mutable launcher/runtime state
• runtime resolved from the project/XDK, not bundled in the plugin
• compatibility with self-hosting composite builds and third-party published-plugin/XDK consumption
• correct behavior for attached, detached, and direct-style execution paths
• clean invalidation across runtime content changes

Open Questions

• Does JVM startup overhead for repeated xtcCompile / testXtc tasks materially dominate self-hosting XDK build time?
• Can a reusable external process provide meaningful wins without making correctness or diagnostics worse?
• What should be fingerprinted for reuse/invalidation?
  • full launcher runtime classpath contents
  • plugin implementation jar
  • relevant JVM args
  • execution mode / launcher kind
• Should compile and test execution share one reusable process, or should they be isolated by launcher role?
• How should crashes, hangs, and protocol incompatibility be detected and recovered from?
• How would future long-lived incremental compiler / type-checker / LSP-style state fit into the chosen model?

Proposed Task List

• Benchmark the current launcher-heavy parts of self-hosting XDK builds.
• Quantify current warm/cold behavior for ATTACHED, DIRECT, and detached-style execution where relevant.
• Identify the concrete launcher task hot spots in the root aggregate/composite build.
• Define the minimal runtime fingerprint needed for safe reuse.
• Prototype a reusable external runtime owner keyed by fingerprint.
• Define a narrow execution protocol for compile/test/run requests.
• Prove that crashes in the reusable process do not poison the Gradle daemon or later builds.
• Verify correctness across branch switches, local code changes, and republished XDK/plugin artifacts.
• Verify mixed execution modes in the same build still invalidate and execute correctly.
• Compare Worker API vs plugin-managed external process tradeoffs.
• Decide whether any of this should become the default for self-hosting XDK builds or remain opt-in.
• Document residual risks and observability requirements before implementation rollout.

Verification Matrix To Build Out

We should eventually verify at least:

• repeated root build / check / publishLocal workflows in the same daemon
• self-hosting XDK compile/test tasks after plugin or javatools changes
• third-party consumer builds against published local and remote artifacts
• switching execution modes between runs
• daemon reuse after crashes or forced worker termination
• branch switches with stale outputs present

Success Criteria

A successful outcome would provide measurable speedup for launcher-heavy self-hosting builds while still guaranteeing:

• no hidden daemon poisoning
• no fake setup tasks that mutate process state outside Gradle inputs/outputs
• reproducible invalidation behavior
• no regressions for external plugin/XDK consumers

Background

This follows the launcher-runtime hardening work that:

• removed loadJavaTools
• removed daemon-global classloader mutation
• introduced explicit launcher runtime resolution
• verified both self-hosting and external-consumer flows

That work made the system safer. This issue asks whether we can now make it faster without giving that safety back.