Run test_multi_process in a fresh subprocess to avoid leaked-thread fork warning#3624
Merged
bdarnell merged 1 commit intoMay 21, 2026
Conversation
When `python3 -m tornado.test` is run in an environment where some test earlier in the suite has left a thread running, the `os.fork()` inside `fork_processes()` triggers `DeprecationWarning: This process (pid=...) is multi-threaded, use of fork() may lead to deadlocks in the child` on Python 3.12+. The test suite turns DeprecationWarnings from tornado into errors, so `test_multi_process` then fails. This has been observed in the Fedora rpm build of tornado on Python 3.15.0b1 (tornadoweb#3623), where it does not reproduce under tox. Rather than chase down every thread leak across the suite, isolate `test_multi_process` so it always starts from a single-threaded state. The actual fork-and-serve logic is moved into a script string that is executed via `python -c` in a fresh interpreter, following the pattern established in autoreload_test for tests that need a clean process. The outer test method just launches the subprocess and asserts a clean exit. PYTHONPATH is propagated so the source tree under test is importable. The script keeps the existing `signal.alarm(5)` timers and `subprocess.run(timeout=30)` is added as a backstop in case the script hangs in a way the alarms don't catch. Tested locally on macOS / Python 3.13 with the full suite plus a deliberately leaked thread before the test to confirm the new isolation holds. The `tearDown` / `get_app` helpers and the `asyncio`, `logging`, and HTTP-related top-level imports are no longer needed and are removed. Fixes tornadoweb#3623
Member
|
Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3623.
fork_processes()ultimately callsos.fork(), which on Python 3.12+ raisesDeprecationWarning: This process (pid=...) is multi-threaded, use of fork() may lead to deadlocks in the childif there is more than one live thread.tornado/test/runtests.pyconfigures DeprecationWarnings emitted fromtornado.*as errors, so any thread that survives an earlier test in the suite will maketest_multi_processfail. The user in #3623 hit this withpython3 -m tornado.teston Python 3.15.0b1 in the Fedora rpm build, but couldn't reproduce it under tox — which matches the maintainer's hypothesis that the difference is which tests get loaded and what global state they leave behind (the asyncio DNS resolver's thread pool being the most likely suspect).This PR takes the route suggested in @bdarnell's comment: move the fork-and-serve body of
test_multi_processinto a small script that runs viapython -cin a fresh interpreter, following the pattern inautoreload_test. The outer test method just launches the subprocess (subprocess.runwithtimeout=30as a backstop on top of the existingsignal.alarm(5)) and asserts a clean exit. Since the subprocess starts single-threaded, whatever the rest of the test suite did to its parent doesn't matter anymore.The script body itself is the same logic that used to live in the test method:
ExpectLogis still active around the fork, the three controlled restarts via/exit=2,/exit=3,/exit=4are preserved, andtask_id()is still asserted in each branch (using bareassertsince we're outside aTestCase).PYTHONPATHis propagated to the subprocess so the source tree under test is importable.The
tearDown/get_apphelpers and theasyncio,logging, and HTTP-related top-level imports were only used by the old in-process version of the test and are removed.SubprocessTestis untouched.Testing
python3 -m tornado.test— 1234 tests, 53 skipped (optional deps), all pass.python3 -m tornado.test tornado.test.process_test.ProcessTest.test_multi_process -v— pass.ProcessTest('test_multi_process')directly. On master this would trip the multi-threaded-fork DeprecationWarning under warnings-as-errors; with this change it passes because the fork happens inside a fresh subprocess.python3 -m flake8 tornado/test/process_test.pyclean;python3 -m black --check tornado/test/process_test.pyclean.I don't have a Python 3.15.0b1 build handy to verify the exact failure-and-fix on the platform from the issue, but the mechanism is the same on 3.12+, and the isolation removes the dependency on the parent's thread state altogether.