LSP Phase 2: warm kotlin-lsp subprocess + per-konstruction workspace.json + stdio bridge

[monkopedia-coder]

Part of #35; Fixes #37. Builds on Phase 1 (#41). Additive + flag-gated (SettingsViewModel.lspEnabled, default OFF): with the flag off (and on CI, which has no engine binary), the editor behaves byte-for-byte as before.

Replaces Phase 1's stub KsrpcLanguageServer with a REAL bridge to a JetBrains kotlin-lsp (intellij-server --stdio) subprocess, producing genuine kcsg-aware diagnostics for the open konstruction. Positions stay in WRAPPED-.kt space for now — the ±3-line csgs↔kotlin translation is the next phase (#38).

Bridge architecture (two LSP legs)

frontend ⇄ (nested ksrpc) ⇄ BridgeLanguageServer ⇄ (stdio LSP / asLspConnection) ⇄ kotlin-lsp
                             \__ pull diagnostics from engine, push up to editor __/

• Frontend↔backend leg (nested ksrpc): the Phase 1 KonstructionService.lsp(client): KsrpcLanguageServer is kept; the editor's KsrpcLanguageClient is stashed.
• Backend↔subprocess leg (dedicated stdio LSP): KotlinLspProcess spawns ONE warm intellij-server --stdio --system-path <dir>, wraps its stdin/stdout with lsp-ksrpc's asLspConnection, and SingleChannelConnection<String>.connectAsLspClient(forwarder) yields a subprocess-facing KsrpcLanguageServer stub. BridgeLanguageServer (a DefaultLanguageServer subclass) delegates each request to that stub. The forwarder is a KsrpcLanguageClient whose textDocumentPublishDiagnostics forwards to the stashed frontend client (with the wrapped-.kt URI rewritten back to the editor's content.csgs URI).
• Single managed subprocess (classpath is fixed); multi-konstruction multiplexing + keep-warm tuning is Phase 5 (LSP Phase 5: harden — lifecycle, warmth, multi-doc, ops; module-extraction decision #40). Phase 2 scopes to the active konstruction.

Pull→push diagnostics (key finding)

kotlin-lsp does not proactively push publishDiagnostics; it answers PULL textDocument/diagnostic (confirmed against the binary: push=None across a full warm cycle). kodemirror's editor client (the Phase 1 design) listens for PUSHED diagnostics. So on didOpen the bridge polls the engine via pull (retrying through the ~120s cold index, then a slow steady-state refresh) and forwards the resulting items up as publishDiagnostics. Any push the engine does send is also forwarded — they coexist.

Per-konstruction setup

KonstructionLspWorkspace synthesizes, under the konstruction's script dir, a workspace dir containing:

• the wrapped .kt via KonstructionControllerImpl.copyContentToScript (konstructor's EXACT compiler wrapping — header swap + footer — so LSP positions match the compiler), and
• a workspace.json mirroring the proven PoC recipe: a JAVA_MODULE with moduleSource + a library dep, and one project-level library pointing at the resolved lib.jar as a single CLASSES root with inclusionOptions: root_itself, lowercase compile scope, inline polymorphic type discriminators. Built with JSON object builders (not @Serializable classes) so the backend needs no serialization compiler plugin.

Lifecycle gating (per the lsp-kotlin maintainer)

LifecycleState is driven by hand: initialized()/shutdown()/exit() → lifecycle.advanceTo(...), and lifecycle.awaitInitialized() gates the first server→client emit (no diagnostics before the client sent initialized). No blocking client-bound request is made from inside initialize() (deadlock risk). On exit the poll is cancelled and the subprocess-facing stub is close()d (cheap leak guard; full teardown/leak test is Phase 5 / #40).

Config / graceful-off

KONSTRUCTOR_KOTLIN_LSP (or -Dkonstructor.kotlinLsp) points at the binary; KONSTRUCTOR_KOTLIN_LSP_SYSTEM_PATH persists/warms the index. If the binary is unset or missing, Config.isKotlinLspAvailable is false, KotlinLspProcess.connect returns null, and BridgeLanguageServer degrades to an inert server (empty capabilities, notifications dropped) — LSP stays off, never a crash. CI has no binary, so this degrades to off.

Also fixes the #41 carry-over nit: EditorPane's produceState now uses catch (_: Exception) (not Throwable).

Verification

Flag-OFF / CI-equivalent — GREEN (JDK 21, clean):

• clean shadowJar — whole project compiles + bundles.
• ./gradlew test — backend + protocol unit/integration tests pass (incl. the new CI-safe KonstructionLspWorkspaceTest asserting the workspace.json matches the strict PoC schema + wrapped-.kt reuse).
• :backend/:frontend/:e2e/:protocol spotlessCheck — clean.
• Full :e2e:test -Pe2e (Xvfb + Mesa-EGL) — all green. LspPipeTest was updated: the Phase-1 canned diagnostic is gone, so with no engine on CI the flag-ON path must establish cleanly and surface zero diagnostics (asserted), and the app stays healthy; the flag-OFF suite is unchanged.

Real-engine — LOCAL ONLY (CI does NOT run the 393MB binary):
Pointed config at /tmp/kotlin-lsp-poc/.../bin/intellij-server (warm --system-path) and ran BridgeLanguageServerIntegrationTest (double-gated on -Dintegration=true + KONSTRUCTOR_KOTLIN_LSP, so it SKIPS on CI). A deliberate-error konstruction (val broken = thisSymbolDoesNotExist123 above a valid cube {} DSL line) drove the real bridge end to end:

REAL-ENGINE diagnostics (1): Unresolved reference 'thisSymbolDoesNotExist123'.

Exactly 1 real kcsg-aware diagnostic, routed to the editor's content.csgs URI, zero false positives on the valid cube {}/primitive {} DSL. This proves: synthesize workspace.json + wrapped .kt → spawn/connect subprocess → initialize/initialized/didOpen → pull diagnostics → push to the frontend client. This engine path is not CI-tested (no binary, no bundled lib resource on CI) — stated explicitly.

🤖 Generated with Claude Code

[monkopedia-reviewer]

Reviewed to full tier-1 standard (correctness + quality + CI-green); approving.

Flag-OFF safety (load-bearing): airtight. EditorPane.produceState returns early on if (!lspEnabled) before any service.lsp() call, so BridgeLanguageServer is never constructed when the flag is off — the editor is byte-for-byte unchanged. Graceful-off (flag ON, no binary = CI): Config.isKotlinLspAvailable gates KotlinLspProcess.connect() → null → delegate=null → initialize() returns empty ServerCapabilities() and every notification handler does delegate ?: return. Inert, never a crash.

Lifecycle: correct. Hand-driven LifecycleState starts at INITIALIZING; advanceTo(INITIALIZED/SHUTTING_DOWN/EXITED) are all legal transitions and advanceTo is the idempotent no-throw variant. awaitInitialized() gates the first push; no blocking client-bound call inside initialize(). exit() cancels the poll + scope Job and exit()/close()s the subprocess stub. The warm subprocess being kept across konstructions is the documented Phase-5/#40 deferral, not a leak.

Tests meaningful. KonstructionLspWorkspaceTest (CI-safe) asserts the strict workspace.json schema (JAVA_MODULE, moduleSource+library deps, lowercase compile, root_itself CLASSES root, lib.jar path) + wrapped-.kt reuse of copyContentToScript. BridgeLanguageServerIntegrationTest genuinely drives the real engine, correctly double-gated and CI-skipped. LspPipeTest updated to assert the inert flag-ON path surfaces zero diagnostics and the app stays healthy.

Quality. Idiomatic Kotlin; dead Phase-1 StubLanguageServer removed with no orphan refs; Apache headers present; scope limited to Phase 2 (no #38 position translation); the #41 Throwable→Exception nit is fixed. Two non-blocking nits: one inline-FQN com.monkopedia.lsp.ServerCapabilities() against the file's import convention, and messages computed a line before the empty-check in the integration test — neither worth blocking.

CI: build-and-test green, MERGEABLE/CLEAN (verified via gh pr checks). Approving and merging.