[BUG] pthread_atfork child handler CUDT::cleanupAtFork deadlocks the child process

[maxtomilov]

Describe the bug
In multi-threaded applications using libsrt 1.5.5, calling fork() to spawn a child process can deadlock the child indefinitely inside the SRT-registered pthread_atfork child handler.

PR #3179 registers srt::CUDT::cleanupAtFork as a pthread_atfork child handler. It calls cleanupAllSockets() -> closeAtFork() -> closeInternal(), which locks ScopedLock(m_ConnectionLock).

fork() only carries the calling thread into the child; mutexes held by other SRT threads at fork time are copied locked-with-no-owner. The child blocks on m_ConnectionLock forever:

  #0  futex_wait (private=0, expected=2, futex_word=0x727a1400a6c0) at ../sysdeps/nptl/futex-internal.h:146
  #1  __GI___lll_lock_wait (futex=futex@entry=0x727a1400a6c0, private=0) at ./nptl/lowlevellock.c:49
  #2  0x0000727a596a0101 in lll_mutex_lock_optimized (mutex=0x727a1400a6c0) at ./nptl/pthread_mutex_lock.c:48
  #3  ___pthread_mutex_lock (mutex=0x727a1400a6c0) at ./nptl/pthread_mutex_lock.c:93
  #4  0x0000727a5991fe15 in __gthread_mutex_lock (__mutex=0x727a1400a6c0) at /usr/include/x86_64-linux-gnu/c++/13/bits/gthr-default.h:749
  #5  std::mutex::lock (this=0x727a1400a6c0) at /usr/include/c++/13/bits/std_mutex.h:113
  #6  std::lock_guard<std::mutex>::lock_guard (__m=..., this=<synthetic pointer>) at /usr/include/c++/13/bits/std_mutex.h:249
  #7  srt::CUDT::closeInternal (this=<optimized out>) at ./external/srt/srt/srtcore/core.cpp:6574
  #8  0x0000727a59920e74 in srt::CUDT::closeAtFork (this=<optimized out>) at ./external/srt/srt/srtcore/core.cpp:6645
  #9  0x0000727a598cba4b in srt::CUDTUnited::cleanupAllSockets (this=this@entry=0x727a599fa720 <srt::CUDT::uglobal()::instance>) at ./external/srt/srt/srtcore/api.cpp:307
  #10 0x0000727a598cbc59 in srt::CUDTUnited::cleanupAtFork (this=this@entry=0x727a599fa720 <srt::CUDT::uglobal()::instance>) at ./external/srt/srt/srtcore/api.cpp:426
  #11 0x0000727a598e0fa5 in srt::CUDT::cleanupAtFork () at ./external/srt/srt/srtcore/api.cpp:3594
  #12 0x0000727a5970e4ab in __run_postfork_handlers (who=who@entry=atfork_run_child, do_locking=do_locking@entry=true, lastrun=lastrun@entry=1) at ./posix/register-atfork.c:187
  #13 0x0000727a596f4147 in __libc_fork () at ./posix/fork.c:109
...

To Reproduce

1. libsrt is configured using "./configure --enable-debug=2 --enable-logging=0 --enable-heavy-logging=0 --enable-apps=0 --enable-c++11 --enable-stdcxx-sync --enable-bonding --enable-maxrexmitbw"
2. Application calls srt_startup(), then creates threads creating SRT sockets with srt_socket() - to listen and to send SRT streams
3. Additional thread is created running sub-process using boost::process::child (using fork internally), the child process hangs forever with the call stack specified above.

Desktop:

• OS: Linux Ubuntu 24.04
• SRT Version / commit ID: 1.5.5
  not reproducible with 1.5.4 and earlier

[ethouris]

I am currently investigating another problem that can be related and I think I have seen this just now. I have prepared a PR for that related problem - #3327 - but I think this thing is not enough. Would you be able to test the version with this fix?

There is also a test for the fork case prepared - it's in examples/fork-test (simple freestanding app). Would you be able to reproduce this with this app?

[maxtomilov]

@ethouris I was not able to reproduce the issue with examples/fork-test apps, but I created a new one based on these two joining server, client and run threads into a single app - and it is reproducible with the new app (attached the source to this comment).

it is also reproducible with the version from #3327 as ScopedLock connectguard(m_ConnectionLock); is still in srt::CUDT::closeInternal.

srt_server+client.c