2026-06-23T15:50:13.0605182Z Fatal Python error: Segmentation fault
2026-06-23T15:50:13.0605688Z
2026-06-23T15:50:13.0606610Z Thread 0x00007637051fb6c0 [ThreadPoolExecu] (most recent call first):
2026-06-23T15:50:13.0607356Z File "/pyenv/versions/3.14.6/lib/python3.14/selectors.py", line 452 in select
2026-06-23T15:50:13.0608604Z File "/pyenv/versions/3.14.6/lib/python3.14/asyncio/base_events.py", line 2018 in _run_once
2026-06-23T15:50:13.0609423Z File "/pyenv/versions/3.14.6/lib/python3.14/asyncio/base_events.py", line 677 in run_forever
2026-06-23T15:50:13.0610306Z File "/pyenv/versions/3.14.6/lib/python3.14/asyncio/base_events.py", line 706 in run_until_complete
2026-06-23T15:50:13.0611068Z File "/pyenv/versions/3.14.6/lib/python3.14/asyncio/runners.py", line 128 in run
2026-06-23T15:50:13.0613214Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 73 in run
2026-06-23T15:50:13.0613974Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 86 in run
2026-06-23T15:50:13.0615091Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 119 in _worker
2026-06-23T15:50:13.0615888Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1024 in run
2026-06-23T15:50:13.0616639Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
2026-06-23T15:50:13.0617363Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1044 in _bootstrap
2026-06-23T15:50:13.0618030Z
2026-06-23T15:50:13.0618472Z Thread 0x00007636461ff6c0 [spmd-executor_4] (most recent call first):
2026-06-23T15:50:13.0619209Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
2026-06-23T15:50:13.0619958Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1024 in run
2026-06-23T15:50:13.0620682Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
2026-06-23T15:50:13.0621408Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1044 in _bootstrap
2026-06-23T15:50:13.0622604Z
2026-06-23T15:50:13.0622998Z Thread 0x00007636a77fe6c0 [spmd-executor_3] (most recent call first):
2026-06-23T15:50:13.0623777Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
2026-06-23T15:50:13.0626730Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1024 in run
2026-06-23T15:50:13.0627423Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
2026-06-23T15:50:13.0628160Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1044 in _bootstrap
2026-06-23T15:50:13.0628658Z
2026-06-23T15:50:13.0629060Z Thread 0x00007636a7fff6c0 [spmd-executor_2] (most recent call first):
2026-06-23T15:50:13.0629803Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
2026-06-23T15:50:13.0630549Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1024 in run
2026-06-23T15:50:13.0631295Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
2026-06-23T15:50:13.0632907Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1044 in _bootstrap
2026-06-23T15:50:13.0633432Z
2026-06-23T15:50:13.0633823Z Thread 0x00007637059fc6c0 [spmd-executor_1] (most recent call first):
2026-06-23T15:50:13.0634717Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
2026-06-23T15:50:13.0635879Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1024 in run
2026-06-23T15:50:13.0636586Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
2026-06-23T15:50:13.0637307Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1044 in _bootstrap
2026-06-23T15:50:13.0637799Z
2026-06-23T15:50:13.0638181Z Thread 0x0000763723fff6c0 [spmd-executor_0] (most recent call first):
2026-06-23T15:50:13.0638922Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/thread.py", line 116 in _worker
2026-06-23T15:50:13.0639662Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1024 in run
2026-06-23T15:50:13.0640492Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1082 in _bootstrap_inner
2026-06-23T15:50:13.0641271Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 1044 in _bootstrap
2026-06-23T15:50:13.0643017Z
2026-06-23T15:50:13.0643403Z Thread 0x0000764c7cc076c0 [python] (most recent call first):
2026-06-23T15:50:13.0644294Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 534 in read
2026-06-23T15:50:13.0645169Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 567 in from_io
2026-06-23T15:50:13.0646040Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
2026-06-23T15:50:13.0646937Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 341 in run
2026-06-23T15:50:13.0647968Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
2026-06-23T15:50:13.0648594Z
2026-06-23T15:50:13.0649018Z Thread 0x0000764c7d854b80 [python] (most recent call first):
2026-06-23T15:50:13.0649661Z File "/pyenv/versions/3.14.6/lib/python3.14/threading.py", line 369 in wait
2026-06-23T15:50:13.0650379Z File "/pyenv/versions/3.14.6/lib/python3.14/concurrent/futures/_base.py", line 449 in result
2026-06-23T15:50:13.0651312Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/engine/core.py", line 473 in execute_ir_on_rank
2026-06-23T15:50:13.0653198Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/engine/core.py", line 699 in evaluate_on_rank
2026-06-23T15:50:13.0654156Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/engine/spmd.py", line 112 in evaluate_pipeline_spmd_mode
2026-06-23T15:50:13.0655039Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/streaming/actor_graph/core.py", line 104 in evaluate_logical_plan
2026-06-23T15:50:13.0656156Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/streaming/parallel.py", line 140 in evaluate_streaming
2026-06-23T15:50:13.0656936Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/callback.py", line 320 in _callback
2026-06-23T15:50:13.0657805Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/polars/_utils/scan.py", line 27 in _execute_from_rust
2026-06-23T15:50:13.0658715Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/polars/lazyframe/frame.py", line 2422 in collect
2026-06-23T15:50:13.0659632Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/polars/lazyframe/opt_flags.py", line 328 in wrapper
2026-06-23T15:50:13.0660540Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/polars/_utils/deprecation.py", line 97 in wrapper
2026-06-23T15:50:13.0661417Z File "/__w/cudf/cudf/python/cudf_polars/cudf_polars/testing/asserts.py", line 90 in assert_gpu_result_equal
2026-06-23T15:50:13.0663420Z File "/__w/cudf/cudf/python/cudf_polars/tests/streaming/test_scan.py", line 86 in test_scan_parquet_use_rapidsmpf_native
2026-06-23T15:50:13.0664515Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
2026-06-23T15:50:13.0665437Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
2026-06-23T15:50:13.0666309Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
2026-06-23T15:50:13.0667147Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
2026-06-23T15:50:13.0667982Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/python.py", line 1720 in runtest
2026-06-23T15:50:13.0668862Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
2026-06-23T15:50:13.0669885Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
2026-06-23T15:50:13.0670779Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
2026-06-23T15:50:13.0671619Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
2026-06-23T15:50:13.0673100Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/runner.py", line 245 in <lambda>
2026-06-23T15:50:13.0673919Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/runner.py", line 353 in from_call
2026-06-23T15:50:13.0674905Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/runner.py", line 244 in call_and_report
2026-06-23T15:50:13.0675802Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/runner.py", line 137 in runtestprotocol
2026-06-23T15:50:13.0677179Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
2026-06-23T15:50:13.0678188Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
2026-06-23T15:50:13.0679118Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
2026-06-23T15:50:13.0680040Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
2026-06-23T15:50:13.0681082Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/xdist/remote.py", line 227 in run_one_test
2026-06-23T15:50:13.0682669Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/xdist/remote.py", line 206 in pytest_runtestloop
2026-06-23T15:50:13.0683641Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
2026-06-23T15:50:13.0684619Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
2026-06-23T15:50:13.0685479Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
2026-06-23T15:50:13.0686342Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/main.py", line 372 in _main
2026-06-23T15:50:13.0687194Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/main.py", line 318 in wrap_session
2026-06-23T15:50:13.0688122Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
2026-06-23T15:50:13.0688977Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_callers.py", line 121 in _multicall
2026-06-23T15:50:13.0689818Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_manager.py", line 120 in _hookexec
2026-06-23T15:50:13.0690768Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/pluggy/_hooks.py", line 512 in __call__
2026-06-23T15:50:13.0692782Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/xdist/remote.py", line 427 in <module>
2026-06-23T15:50:13.0693645Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 1291 in executetask
2026-06-23T15:50:13.0694751Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 341 in run
2026-06-23T15:50:13.0695676Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
2026-06-23T15:50:13.0696668Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
2026-06-23T15:50:13.0697582Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 1273 in serve
2026-06-23T15:50:13.0698456Z File "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/execnet/gateway_base.py", line 1806 in serve
2026-06-23T15:50:13.0699140Z File "<string>", line 8 in <module>
2026-06-23T15:50:13.0699775Z File "<string>", line 1 in <module>
2026-06-23T15:50:13.0700163Z
2026-06-23T15:50:13.0700571Z Current thread's C stack trace (most recent call first):
2026-06-23T15:50:13.0701320Z Binary file "/pyenv/versions/3.14.6/lib/libpython3.14.so.1.0", at _Py_DumpStack+0x45 [0x764c7de548d5]
2026-06-23T15:50:13.0703186Z Binary file "/pyenv/versions/3.14.6/lib/libpython3.14.so.1.0", at +0x312f6c [0x764c7de6af6c]
2026-06-23T15:50:13.0703895Z Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x45330 [0x764c7d984330]
2026-06-23T15:50:13.0704993Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/libcudf_streaming/lib64/libcudf_streaming.so", at _ZN4cudf2io22parquet_reader_optionsD1Ev+0xf8 [0x764bedeaab98]
2026-06-23T15:50:13.0706161Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/libcudf_streaming/lib64/libcudf_streaming.so", at +0x5c4ac [0x764bedea64ac]
2026-06-23T15:50:13.0707508Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/rapidsmpf/streaming/core/actor.abi3.so", at +0x1b5ce [0x764bedab05ce]
2026-06-23T15:50:13.0708712Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/rapidsmpf/streaming/core/actor.abi3.so", at _ZNSt6vectorIN4coro6detail13when_all_taskIvEESaIS3_EED1Ev+0x32 [0x764bedab3002]
2026-06-23T15:50:13.0710047Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/rapidsmpf/streaming/core/actor.abi3.so", at +0xcca7 [0x764bedaa1ca7]
2026-06-23T15:50:13.0711092Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/rapidsmpf/streaming/core/actor.abi3.so", at +0x1b545 [0x764bedab0545]
2026-06-23T15:50:13.0712927Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/librapidsmpf/lib64/librapidsmpf.so", at +0x15fd23 [0x764bfdd5fd23]
2026-06-23T15:50:13.0713953Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/librapidsmpf/lib64/librapidsmpf.so", at +0x1616af [0x764bfdd616af]
2026-06-23T15:50:13.0715199Z Binary file "/__w/cudf/cudf/venv_polars_1.35/lib/python3.14/site-packages/librapidsmpf/lib64/librapidsmpf.so", at _ZN4coro11thread_pool8executorEm+0xdc [0x764bfdd9166c]
2026-06-23T15:50:13.0716140Z Binary file "/lib/x86_64-linux-gnu/libstdc++.so.6", at +0xecdb4 [0x764c6c26edb4]
2026-06-23T15:50:13.0716836Z Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x9caa4 [0x764c7d9dbaa4]
2026-06-23T15:50:13.0717523Z Binary file "/lib/x86_64-linux-gnu/libc.so.6", at __clone+0x44 [0x764c7da68a64]
CUDF_POLARS__EXECUTOR__MAX_IO_THREADS=32 CUDA_VISIBLE_DEVICES=0 pytest -vs python/cudf_polars/tests/streaming/test_scan.py -k 'use_rapidsmpf_native[spmd-small' -n 4 --count 100
Edit: this might be related to that PR, which is changing the ABI of
SourceInfo. I'm not sure how that'd lead to a segfault yet.https://github.com/rapidsai/cudf/actions/runs/28035236212/job/82994526731?pr=22739 is a PR with an unrelated failure in
python/cudf_polars/tests/streaming/test_scan.py::test_scan_parquet_use_rapidsmpf_native. Here's the output from pytest:I've been unable to reproduce locally using
pytest-repeatwith a high--count, and a high number of IO threads: