Skip to content

Fix Electrum hanging on shutdown: cancel-safe, resumable announcement proof-of-work - #2

Merged
nothing-stops-this-train merged 1 commit into
mainfrom
fix/shutdown-hang-pow-deadlock
Jul 28, 2026
Merged

Fix Electrum hanging on shutdown: cancel-safe, resumable announcement proof-of-work#2
nothing-stops-this-train merged 1 commit into
mainfrom
fix/shutdown-hang-pow-deadlock

Conversation

@nothing-stops-this-train

Copy link
Copy Markdown
Owner

The bug

Electrum never fully shut down once the swap server had been enabled — the window closed but the process stayed alive.

Electrum's asyncio loop runs on a non-daemon thread (EventLoop, electrum/util.py:1710), so anything that blocks that thread keeps the interpreter alive forever. Shutdown also goes through asyncio.run_coroutine_threadsafe(self.stop(), ...).result()blocking, no timeout (electrum/daemon.py:694) — so a wedged loop freezes the GUI thread too.

Root cause

electrum.util.gen_nostr_ann_pow deadlocks the loop thread when the awaiting task is cancelled mid-grind:

with ProcessPoolExecutor(max_workers=max_workers) as executor:   # __exit__ -> shutdown(wait=True)
    ...
    done, pending = await asyncio.wait(tasks, return_when=FIRST_COMPLETED)  # CancelledError here
    executor.shutdown(wait=False, cancel_futures=True)                     # success path ONLY

On cancellation the context manager's __exit__ runs executor.shutdown(wait=True) — a synchronous join, on the event loop thread. The workers (util.py:2381) only exit when a shared shutdown Event is set, and that Event is only set by a worker that finds a solution. The join never returns.

This plugin reached that code from two tasks it cancels at shutdown: run_nostr_server and _one_shot_publish both begin with set_nostr_proof_of_work, and stop_server() cancels both from the close_wallet / on_close_window hooks.

SWAPSERVER_POW_TARGET defaults to 30 bits and SWAPSERVER_ANN_POW_NONCE to 0 (which scores 0 bits), so a fresh server always grinds — measured ~3.5 min expected on an 8-core box, and exponentially distributed, so 8–10 min is unremarkable. Quitting anywhere in that window wedged the loop permanently. The two tasks also ground concurrently, each spawning cpu_count-1 workers plus a Manager process, doubling CPU use for identical work.

Verified before/after, against the real upstream function

Same harness, cancel a grind after 4s, then probe whether the loop can still run asyncio.sleep(0):

Loop after cancel EventLoop thread Process
upstream gen_nostr_ann_pow WEDGED still alive SIGKILL required (137)
this PR's pow.grind OK exited clean exit (0)

The fix

Own the proof of work (plugins/swapserver_gui/pow.py) and seed the nonce before starting the nostr transport, so upstream's set_nostr_proof_of_work always short-circuits on a cached nonce and never enters the broken function.

  • Raw multiprocessing.Process instead of ProcessPoolExecutor, whose atexit hook joins workers at interpreter shutdown — itself a hang vector. Cancellation signals, terminate()s, and hands joining to a daemon thread, so it never blocks the loop or the GUI thread.
  • Resumable. Workers checkpoint their nonce cursor; it's persisted per lane in a new config var. There is no "partly done" with a PoW — each nonce is an independent 2^-target trial — but the scanned range is worth keeping: upstream restarts every worker from a deterministic start nonce, so a cancelled grind re-tests exactly what it already rejected. A user restarting Electrum every few minutes might never finish a 30-bit PoW. Now work accumulates monotonically.
  • Tracking the best nonce seen keeps cursors valid across any target change: if the best is below the new target, nothing in the scanned range meets it either. It also lets a lowered target succeed instantly.
  • Thread fallback where fork is unavailable (Windows/Android). The plugin package isn't guaranteed importable in a spawned child — it may load from a zip as electrum_external_plugins.swapserver_gui — so a process backend there would fail at unpickling. Single-core but correct and cancel-safe; CI's 2-core runners exercise this path.

Also fixed (contributing to unclean teardown)

  • stop_server() unregisters the HTTP EventListener synchronously — the cleanup coroutine was fire-and-forget and could be cancelled by the loop shutting down, leaving stale callbacks registered against a stopped wallet.
  • The 4s refresh QTimer is now stopped when the tab is removed; it kept calling into get_asyncio_loop() while the loop was being torn down.
  • publish_now waits on a PoW gate rather than computing its own, so an offer is never announced with a nonce takers would reject.

UI

The tab shows proof-of-work progress (best bits so far / hashes scanned — no fake percentage, since the search is memoryless) and an estimated grind time next to the target, measured on the local machine. Raising the target past 30 bits now asks for confirmation: 35 bits is ~2 hours and 40 bits ~2.5 days on an 8-core box.

Testing

52 tests pass under unittest discover, as CI runs.

tests/test_shutdown_e2e.py reproduces the reported bug end to end, driving the real plugin through Electrum's actual shutdown sequence (non-daemon EventLoop thread, stopping_fut, cancel-everything teardown):

  • the wallet closes with the loop still responsive while grinding
  • stop_server() doesn't block the GUI thread (< 2s)
  • the non-daemon loop thread actually exits
  • the blocking .result() with no timeout from daemon.py:694 returns — the precise call that hangs in the field
  • shutdown works even if the plugin hooks never fire at all
  • a whole subprocess interpreter exits mid-grind, leaving no orphaned workers
  • PoW progress is persisted across the shutdown

tests/test_pow.py covers cancel-safety on both backends, worker reaping, resume-past-cursor, state invalidation on wallet switch, and rejection of hand-edited/corrupt state.

Known upstream issue, not addressed here

NostrTransport.__exit__ (submarine_swaps.py:1798) blocks the loop ~5s on every cancel via fut.result(timeout=5) called from the loop thread. Bounded and no longer fatal once the deadlock is gone; documented in a comment rather than worked around.

🤖 Generated with Claude Code

https://claude.ai/code/session_013zETFC6MhnsCuxvQDGxqV5

… PoW

Electrum never fully shut down once the swap server had been enabled. The
asyncio loop runs on a non-daemon thread ('EventLoop', electrum/util.py), so
anything that blocks that thread keeps the interpreter alive forever.

Root cause: electrum.util.gen_nostr_ann_pow deadlocks the loop thread when the
awaiting task is cancelled mid-grind.

    with ProcessPoolExecutor(max_workers=max_workers) as executor:
        ...
        done, pending = await asyncio.wait(tasks, ...)      # CancelledError
        executor.shutdown(wait=False, cancel_futures=True)  # success path only

On cancellation the context manager's __exit__ runs executor.shutdown(wait=True),
a synchronous join, on the loop thread. The workers only exit when a shared
shutdown Event is set, and that Event is only set by a worker that *finds* a
solution, so the join never returns.

The plugin reached that code from two tasks it cancels at shutdown:
run_nostr_server and _one_shot_publish both begin with set_nostr_proof_of_work.
stop_server() cancels both from the close_wallet / on_close_window hooks. With
the default 30-bit target and nonce 0 a fresh server always grinds (~3.5 min on
an 8-core box), so quitting in that window wedged the loop permanently. The two
tasks also ground concurrently, doubling CPU use for identical work.

Fix: own the proof of work (plugins/swapserver_gui/pow.py) and seed the nonce
before starting the nostr transport, so upstream's set_nostr_proof_of_work
always short-circuits on a cached nonce and never enters the broken function.

  * raw multiprocessing.Process instead of ProcessPoolExecutor, whose atexit
    hook is itself a hang vector; cancellation signals, terminates and hands
    joining to a daemon thread, never blocking the loop or the GUI thread
  * resumable: workers checkpoint their nonce cursor, persisted per lane in a
    new config var, so a grind cancelled by quitting continues into untested
    nonce space instead of re-scanning the range upstream deterministically
    re-tests. Tracking the best nonce seen keeps cursors valid for any target
  * thread fallback where fork is unavailable (Windows/Android), since the
    plugin package is not guaranteed importable in a spawned child

Also fixed, all contributing to an unclean teardown:
  * stop_server() now unregisters the HTTP EventListener synchronously, so a
    cancelled cleanup coroutine cannot leave stale callbacks registered
  * the 4s refresh QTimer is stopped when the tab is removed; it was calling
    into get_asyncio_loop() while the loop was being torn down
  * publish_now waits on a PoW gate rather than computing its own

The tab now shows proof-of-work progress and an estimated grind time next to
the target, and confirms before raising the target past 30 bits.

Tests: 52 pass. tests/test_shutdown_e2e.py reproduces the reported bug end to
end, including a subprocess test asserting a whole interpreter actually exits
while the proof of work is still running, and that the wallet closes cleanly
mid-grind with progress preserved. Verified against the real upstream helper:
cancelling gen_nostr_ann_pow wedges the loop and the process needs SIGKILL,
while the replacement exits cleanly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013zETFC6MhnsCuxvQDGxqV5
@nothing-stops-this-train
nothing-stops-this-train merged commit caa4c78 into main Jul 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant