Segfault in OutboundPort::notify during evalExprWithProgress (Signal 11, libLLVM)

[lightx]

Describe the bug

nixd segfaults (Signal 11/SEGV) during expression evaluation while sending a WorkDoneProgressEnd LSP notification. The crash occurs in llvm::json::isUTF8(llvm::StringRef) called from lspserver::OutboundPort::notify() — the StringRef appears to reference invalid memory (address 0x59d44836c755 is clearly not a valid pointer), causing a null-pointer-adjacent dereference inside libLLVM.so.21.1.

Stack trace (crashing thread 1671726)

#0  0x000078e2986463d9 llvm::json::isUTF8(llvm::StringRef, unsigned long*) (libLLVM.so.21.1 + 0xc463d9)
#1  0x000078e2a1c833c0 lspserver::OutboundPort::notify(llvm::StringRef, llvm::json::Value) (libnixd-lspserver.so + 0x213c0)
#2  0x000078e2a1de21c8 llvm::detail::UniqueFunctionBase<...>::CallImpl<lspserver::LSPServer::mkOutNotifiction<WorkDoneProgressEnd>...> (libnixd.so + 0x1141c8)
#3  0x000078e2a1db20f2 llvm::detail::UniqueFunctionBase<...>::CallImpl<nixd::Controller::evalExprWithProgress(...)...> (libnixd.so + 0xe40f2)
#4  0x000078e2a1de6dcf llvm::detail::UniqueFunctionBase<...>::CallImpl<lspserver::LSPServer::mkOutMethod<...>...> (libnixd.so + 0x118dcf)
#5  0x000078e2a1c8c19d lspserver::LSPServer::onReply(llvm::json::Value, llvm::Expected<llvm::json::Value>) (libnixd-lspserver.so + 0x2a19d)
#6  0x000078e2a1c8320b lspserver::InboundPort::dispatch(llvm::json::Value, lspserver::MessageHandler&) (libnixd-lspserver.so + 0x2120b)
#7  0x000078e2a1c851a9 lspserver::InboundPort::loop(lspserver::MessageHandler&) (libnixd-lspserver.so + 0x231a9)

The main thread (1671677) was in Controller::~Controller() → _M_erase (destroying AttrSetClientProc map entries), waiting on thread join — indicating nixd was shutting down while the eval thread was still trying to send progress notifications through a dying/dead channel.

Kernel log

May 24 17:39:41 hornet kernel: nixd[1671726]: segfault at 59d44836c755 ip 000078e2986463d9 sp 000078e2799ce340 error 4 in libLLVM.so.21.1[c463d9,78e2983b0000+5bd0000] likely on CPU 10 (core 2, socket 0)
May 24 17:39:41 hornet kernel: Code: 25 28 00 00 00 48 89 45 f8 31 c0 48 39 fe 74 41 48 89 f9 48 89 f8 eb 12 66 0f 1f 84 00 00 00 00 00 48 83 c0 01 48 39 c6 74 27 <80> 38 00 79 f2 48 8d 7d f0 48 89 55 e0 48 89 4d f0 48 89 4d e8 e8

The <80> 38 00 instruction is cmp byte [rax], 0 — classic strlen-style scan hitting an invalid pointer.

Configuration

{
  "nixd": {
    "formatting": { "command": ["nixfmt"] }
  }
}

Running inside Neovim via nvim-lspconfig.

To Reproduce

Not reliably reproducible — this is an intermittent crash that occurs during heavy flake evaluation. The flake in question (nix flake check) passes cleanly. This appears to be a race condition where the progress notification callback runs after or during controller teardown.

Steps that may trigger it:

1. Open a large NixOS flake in an editor with nixd LSP
2. Wait for nixd to begin evaluating (progress notifications fire)
3. Crash occurs when the LSP channel disconnects or the controller starts shutting down while eval callbacks are still pending

Expected behavior

nixd should not segfault. Progress notifications should be safely dropped if the outbound channel is no longer valid.

Additional context

• nixd version: 2.9.1
• NixOS 25.05 (Warbler), Linux 6.18.26
• x86_64
• Nix store path: /nix/store/nq6rgzv6kxs01jncg3b19fdygfnxkjwh-nixd-2.9.1
• No custom nixd config beyond formatting command
• Flakes evaluation with home-manager integration (7 hosts)

[inclyc]

Hi @lightx, thank you very much for the detailed crash report with your own careful analysis!

To address this issue I need to confirm and reproduce the crash, but currently I don't know how to get started. The main problem is that you mentioned a "heavy" flake that triggers this issue — I'm not sure what counts as "heavy". Do you have an actual example (that can be cloned even it is "heavy") ? This kind of crash might actually be reducible to a smaller flake that contains certain specific fragments (try it?). If you could give me a more detailed/reliable reproduction method, it would be much easier to define the problem!

indicating nixd was shutting down while the eval thread was still trying to send progress notifications through a dying/dead channel.

I noticed this error happens while nixd is shutting down. In theory, an LSP server should not shut down on its own without the user actively closing the editor or restarting. Do I need to manually shut it down when reproducing?

[lightx]

Hi @inclyc

Here's a minimal self-contained flake that reproduces the eval load: github.com/lightx/nixd-repro

It has the same input graph as my full config (nixpkgs + home-manager + hermes-agent + uv2nix/pyproject-nix chain + lazyvim + zen-browser + claude-code + agenix + disko + nh) but outputs = _: {} — no hosts defined, just the inputs. nixd still crashes when opening flake.nix in neovim because it evaluates the full input tree in the background.

To confirm your question: nixd exited on its own — I did not close neovim or restart the LSP. The crash happened mid-evaluation with no user action. Let me do more research on this. Something is off…

[lightx]

Update — root cause found, automated reproducer with 100% crash rate

I dug deeper. The repo at github.com/lightx/nixd-repro now has:

• flake.nix — minimal self-contained reproduction flake (same input graph, outputs = _: {})
• test_race.py — automated test, run with python3 test_race.py
• race-condition-analysis.md — full analysis with coredump evidence

Running the test:

git clone https://github.com/lightx/nixd-repro
cd nixd-repro
python3 test_race.py

Result on my machine (nixd 2.9.1, x86_64-linux): 3/3 SIGSEGV. No editor needed — the script drives nixd directly over LSP stdio.

---

The actual trigger

My earlier statement that "nixd exited on its own" was misleading — I was watching it crash while an AI agent was editing .nix files in the background, causing neovim to repeatedly restart nixd. The real trigger is abrupt stdin close (EOF without the shutdown+exit handshake). This happens when the editor crashes, is killed, or restarts nixd without a graceful disconnect. Clean shutdown+exit does not hit the race.

---

Two threads caught mid-race (from coredumpctl):

Thread A (eval, crashing):
  InboundPort::loop() → dispatch() → onReply()
  → evalExprWithProgress callback
  → EndWorkDoneProgress → Out->notify()        ← Out already freed
  → isUTF8(StringRef)                          ← SEGV

Thread B (main, simultaneously):
  main() → ~Controller() → _Rb_tree::_M_erase
  → std::thread::join()                        ← blocked waiting

Root cause: EndWorkDoneProgress captures Out.get() as a raw pointer (via mkOutNotifiction in Support.cpp). ~Controller() frees Out before joining the AttrSetClient Input threads. A still-running Input thread fires the callback on the freed pointer.

Suggested fixes (details in race-condition-analysis.md):

• Option A (minimal): Guard Out->notify() with if (ReceivedShutdown) return; in endWorkDoneProgress
• Option B (safer): Replace the raw Out.get() capture in EndWorkDoneProgress with a weak_ptr<OutboundPort>

---

Does this give you enough to work with? Happy to test any patch.

Troubleshooting and analysis assisted by Claude Sonnet (claude.ai/code) and GLM-5.1.

[inclyc]

My earlier statement that "nixd exited on its own" was misleading — I was watching it crash while an AI agent was editing .nix files in the background, causing neovim to repeatedly restart nixd. The real trigger is abrupt stdin close (EOF without the shutdown+exit handshake). This happens when the editor crashes, is killed, or restarts nixd without a graceful disconnect. Clean shutdown+exit does not hit the race.

If that's the case, I think maybe we shouldn't fix nixd itself. The "AI agent" should be fixed instead—after all, nixd probably wasn't designed with the expectation that its stdin would be abruptly closed. However, since your proposed change is very simple, we could introduce it as a workaround to avoid causing trouble for users. The next step would be to evaluate how many users would actually benefit from this change and whether it's worth it. Is this problem caused by the AI agent affecting a wide range of users, or is it limited to just a few individuals (like you, who might have been particularly unlucky)?

[lightx]

Unsure if its affecting "enough" users to warrant a fix. Might be worth it as a defensive measure regardless of the root cause.