RSEQ related crashes since linux-6.19

[sakaru]

What

The unit test //tcmalloc/testing:background_test_no_glibc_rseq reliably fails since the linux commit 39a167560a61 ("rseq: Optimize event setting").

This was discussed on this archlinux issue. It was also discussed on the kernel mailing list. The latter thread also says:

Is this using tcmalloc? If so, that is somewhat expected because
tcmalloc is known to violate upstream rseq ABI. IIRC you should get a
nice splat if you enable rseq debug mode (echo 1 > /debug/rseq/debug).

Perhaps this is the nudge Google needs to go fix this.

Failing unit test:

[arch@podman-mongodb tcmalloc]$ GLIBC_TUNABLES=glibc.pthread.rseq=1 bazel-bin/tcmalloc/testing/background_test_no_glibc_rseq
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from BackgroundTest
[ RUN      ] BackgroundTest.Defaults
[       OK ] BackgroundTest.Defaults (0 ms)
[ RUN      ] BackgroundTest.Stress
[       OK ] BackgroundTest.Stress (10017 ms)
[----------] 2 tests from BackgroundTest (10018 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (10018 ms total)
[  PASSED  ] 2 tests.

[arch@podman-mongodb tcmalloc]$ GLIBC_TUNABLES=glibc.pthread.rseq=0 bazel-bin/tcmalloc/testing/background_test_no_glibc_rseq
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from BackgroundTest
[ RUN      ] BackgroundTest.Defaults
[       OK ] BackgroundTest.Defaults (0 ms)
[ RUN      ] BackgroundTest.Stress
14207 ./tcmalloc/internal/percpu_tcmalloc.h:852] CHECK in Pop: next (false)
Aborted                    (core dumped) GLIBC_TUNABLES=glibc.pthread.rseq=0 bazel-bin/tcmalloc/testing/background_test_no_glibc_rseq

[arch@podman-mongodb tcmalloc]$ GLIBC_TUNABLES=glibc.pthread.rseq=0 /tmp/bazel test //tcmalloc/testing:background_test_no_glibc_rseq
WARNING: Build option --test_env has changed, discarding analysis cache (this can be expensive, see https://bazel.build/advanced/performance/iteration-speed).
INFO: Analyzed target //tcmalloc/testing:background_test_no_glibc_rseq (0 packages loaded, 5304 targets configured).
FAIL: //tcmalloc/testing:background_test_no_glibc_rseq (Aborted) (see /home/arch/.cache/bazel/_bazel_arch/3c882abdfeb0a7dc4942cd1a7209754a/execroot/_main/bazel-out/k8-fastbuild/testlogs/tcmalloc/testing/background_test_no_glibc_rseq/test.log)
INFO: Found 1 test target...
Target //tcmalloc/testing:background_test_no_glibc_rseq up-to-date:
  bazel-bin/tcmalloc/testing/background_test_no_glibc_rseq
INFO: Elapsed time: 3.948s, Critical Path: 1.62s
INFO: 2 processes: 644 action cache hit, 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 2 total actions
//tcmalloc/testing:background_test_no_glibc_rseq                         FAILED in 1.6s
  /home/arch/.cache/bazel/_bazel_arch/3c882abdfeb0a7dc4942cd1a7209754a/execroot/_main/bazel-out/k8-fastbuild/testlogs/tcmalloc/testing/background_test_no_glibc_rseq/test.log

Executed 1 out of 1 test: 1 fails locally.


[arch@podman-mongodb tcmalloc]$ cat /home/arch/.cache/bazel/_bazel_arch/3c882abdfeb0a7dc4942cd1a7209754a/execroot/_main/bazel-out/k8-fastbuild/testlogs/tcmalloc/testing/background_test_no_glibc_rseq/test.log
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //tcmalloc/testing:background_test_no_glibc_rseq
-----------------------------------------------------------------------------
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from BackgroundTest
[ RUN      ] BackgroundTest.Defaults
[       OK ] BackgroundTest.Defaults (0 ms)
[ RUN      ] BackgroundTest.Stress
28 ./tcmalloc/internal/percpu_tcmalloc.h:852] CHECK in Pop: next (false)
*** SIGABRT received at time=1773167104 on cpu 1 ***
23 ./tcmalloc/internal/percpu_tcmalloc.h:852] CHECK in Pop: next (false)
[failure_signal_handler.cc : 377] RAW: Signal 6 raised at PC=0x7f7d81d89a2c while already in AbslFailureSignalHandler()
19 ./tcmalloc/internal/percpu_tcmalloc.h:852] CHECK in Pop: next (false)
[failure_signal_handler.cc : 377] RAW: Signal 6 raised at PC=0x7f7d81d89a2c while already in AbslFailureSignalHandler()
[failure_signal_handler.cc : 377] RAW: Signal 11 raised at PC=0x5580fdd87519 while already in AbslFailureSignalHandler()
[failure_signal_handler.cc : 377] RAW: Signal 11 raised at PC=0x5580fdd87519 while already in AbslFailureSignalHandler()
20 ./tcmalloc/internal/percpu_tcmalloc.h:852] CHECK in Pop: next (false)
[failure_signal_handler.cc : 377] RAW: Signal 6 raised at PC=0x7f7d81d89a2c while already in AbslFailureSignalHandler()
PC: @     0x7f7d81d89a2c  (unknown)  (unknown)
    @     0x7f7d82ae1c35         64  absl::lts_20260107::WriteFailureInfo()
    @     0x7f7d82ae1e74         96  absl::lts_20260107::AbslFailureSignalHandler()
    @     0x7f7d81d2f2d0       3120  (unknown)
    @     0x7f7d81d2f1a0         32  raise
    @     0x7f7d81d165fe        192  abort
    @     0x7f7d826b2d8d        704  tcmalloc::tcmalloc_internal::Crash()
    @     0x7f7d826b2dc2         48  tcmalloc::tcmalloc_internal::CheckFailed()
    @     0x5580fdc850a2        640  tcmalloc::tcmalloc_internal::CheckFailed<>()
    @     0x5580fdd78ab6        368  operator new()
    @     0x5580fdb7bc44         64  std::__new_allocator<>::allocate()
    @     0x5580fdb7a3a3         48  std::_Vector_base<>::_M_allocate()
    @     0x5580fdb7a5d1        256  std::vector<>::_M_realloc_append<>()
    @     0x5580fdb7934a        128  std::vector<>::emplace_back<>()
    @     0x5580fdb77c6d         48  std::vector<>::push_back()
    @     0x5580fdb76395        208  tcmalloc::AllocatorHarness::Run()
    @     0x5580fdb73d9d         32  tcmalloc::(anonymous namespace)::BackgroundTest_Stress_Test::TestBody()::{lambda()#1}::operator()()
    @     0x5580fdb74466         48  std::__invoke_impl<>()
    @     0x5580fdb74354         48  std::__invoke_r<>()
    @     0x5580fdb7421b         64  std::_Function_handler<>::_M_invoke()
    @     0x5580fdb78128         32  std::function<>::operator()()
    @     0x5580fdb76b50         32  tcmalloc::ThreadManager::Start()::{lambda()#1}::operator()()
    @     0x5580fdb7eba2         48  std::__invoke_impl<>()
    @     0x5580fdb7eaf2         48  std::__invoke<>()
    @     0x5580fdb7ea49         64  std::thread::_Invoker<>::_M_invoke<>()
    @     0x5580fdb7e9e6         32  std::thread::_Invoker<>::operator()()
    @     0x5580fdb7e9aa         32  std::thread::_State_impl<>::_M_run()
    @     0x7f7d820e55a4         32  (unknown)
    @     0x7f7d81d8797a        224  (unknown)
    @     0x7f7d81e0b2bc  (unknown)  (unknown)

[maratik123]

Let me left it here: https://lkml.org/lkml/2012/12/23/75

[sakaru]

Let me left it here: https://lkml.org/lkml/2012/12/23/75

I think I already addressed this by quoting the mailing list. tcmalloc (reportedly, I'm no expert) does not abide by the rseq ABI, thus the "WE DO NOT BREAK USERSPACE!" does not apply.

[ckennelly]

TCMalloc has had a per-CPU slab caching optimization since around aaa4fbf in September 2023. It shortens the critical path of malloc operations, and finding an alternative that preserves the performance benefits may prove challenging.

There was a prior discussion regarding the slab caching optimization in #144, fwiw.

[dvyukov]

This test fails for me on both v6.17 and v6.6 kernels in the same way...

[lvoytek]

Hello, I am working on the Ubuntu 26.04 release with kernel 7.0 and was pinged about this today. We were wondering what the tcmalloc maintainers believe the impact of this may be. We currently use tcmalloc for several applications in the Ubuntu archive, including MySQL 8.4 since 25.10 (kernel 6.17). I experimented with MySQL on the 26.04 beta today with kernel 7.0 and did not hit any issues. Could you confirm if we should be concerned about this issue outside of MongoDB, or just consider this to be a normal bug to track for a fix later on?

[ckennelly]

It appears that torvalds/linux@39a1675 changed the behavior of the membarrier syscall.

Previously, membarrier caused the kernel to overwrite fields like __rseq_abi.cpu_id and __rseq_abi.cpu_id_start.

Since that change, membarrier alone is not sufficient, which breaks our cross-CPU fencing operations.

We are looking at potential fixes that can be done entirely inside TCMalloc, but this is an upstream change that broke userspace.

[askovpen]

Yesterday released Ubuntu 26.04 LTS with kernel 7.0. All software, uses this lib will not work

[sakaru]

Yesterday released Ubuntu 26.04 LTS with kernel 7.0. All software, uses this lib will not work

I realise you are likely not the one who wrote the MongoDB release notes. I do want to stress something I included in the initial post though; setting GLIBC_TUNABLES=glibc.pthread.rseq=1 is a workaround, though I don't know what other consequences that has (perhaps performance regressions).

I installed Ubuntu 26.04 just now, and MongoDB is not packaged in the repositories, but I tried running it via docker. Running the latest tags fails because of a hardcoded kernel version check¹, but older containers which don't have that restriction work with that tunable set.

$ lsb_release -d
Description:    Ubuntu 26.04 LTS

$ sudo docker run --rm docker.io/mongodb/mongodb-community-server
ERROR: Detected Linux kernel 7.0.0-14-generic. MongoDB 8.0+ utilizes the tcmalloc allocator which has a known issue with the v6.19 and newer Linux kernel. This container will not start by default on v6.19+.

$ sudo docker run --rm docker.io/mongodb/mongodb-community-server:8.0.19-ubi9
# prints stuff, then after 30 seconds crashes

$ sudo docker run --env GLIBC_TUNABLES=glibc.pthread.rseq=1 --rm docker.io/mongodb/mongodb-community-server:8.0.19-ubi9
# works

So MongoDB would work with that environment variable set. Again with the caveat that I don't know what other consequences that entails.

Footnotes

1. https://github.com/mongodb/mongo/blob/cd90f28f0a69ab94a913fb5310a69a7602e2f107/src/mongo/db/startup_check_rseq.cpp#L97-L107 ↩

[dvyukov]

FTR potential fixes or revert is discussed on the kernel mailing lists:
https://lore.kernel.org/all/CAHnCjA25b+nO2n5CeifknSKHssJpPrjnf+dtr7UgzRw4Zgu=oA@mail.gmail.com/

[askovpen]

latest mongodb stable release:

$ sudo docker run --env GLIBC_TUNABLES=glibc.pthread.rseq=1 --rm docker.io/mongodb/mongodb-community-server:8.2.7-ubi9
ERROR: Detected Linux kernel 7.0.0-14-generic. MongoDB 8.0+ utilizes the tcmalloc allocator which has a known issue with the v6.19 and newer Linux kernel. This container will not start by default on v6.19+.
[

[antonio-giovannetti]

Hello, are you planning to fix the issue? It blocks upgrade to mongodb 8, see issue

[ckennelly]

There's a patch series out for fixing this in the kernel: https://lkml.org/lkml/2026/4/29/73

6.19 introduced a number of regressions: membarrier trigger republish of cpu_id_start, lack of mutual exclusion on AArch64, and some more minor ones (setting cpu_id_start=0 on unregister)