SIGBUS on uninstall after nixpkgs bump

[grahamc]

After bumping nixpkgs, the uninstaller is crashing with a SIGBUS.

• Good: =0.1.698755+rev-807e9154dcb16384b1b765ebe9cd2bba2ac287fd
• Bad: >=0.1.702572+rev-7ffd9ae656aec493492b44d0ddfb28e79a1ea25d

The most glaringly notable change between these two versions is the introduction of how Apple SDKs are packaged: NixOS/nixpkgs#346043.

The issue can be reproduced on:

• MacOS 13.7
• MacOS 14.6.1
• MacOS 14.7.1
• MacOS 15.1

by:

1. Build nix-installer using a version of Nixpkgs newer than 0.1.698755+rev-807e9154dcb16384b1b765ebe9cd2bba2ac287fd.
2. Install Nix: ./nix-installer install --no-confirm
3. Run nix run nixpkgs#hello -- Note: this step is, surprisingly, load bearing!
4. Uninstall Nix /nix/nix-installer uninstall --no-confirm
5. Observe the removal pretty much completes, but fails with SIGBUS

Using lldb's bt all, we can see the main thread is using security_framework and rustls_native_certs to get certificates:

  thread #1, name = 'main', queue = �[32m'com.apple.main-thread'�[0m
    frame #0: �[33m0x000000019d256bac�[0m libsystem_kernel.dylib`__ulock_wait + 8
    frame #1: �[33m0x000000019d0e5dd8�[0m libdispatch.dylib`_dlock_wait + 56
    frame #2: �[33m0x000000019d0e5ad4�[0m libdispatch.dylib`_dispatch_wait_on_address + 140
    frame #3: �[33m0x000000019d0e61a4�[0m libdispatch.dylib`_dispatch_group_wait_slow + 56
    frame #4: �[33m0x000000019d0e9394�[0m libdispatch.dylib`dispatch_block_wait + 284
    frame #5: �[33m0x00000001a02deb3c�[0m Security`SecTrustSettingsCopyCertificates + 244
    frame #6: �[33m0x0000000100dca798�[0m ni3`security_framework::trust_settings::TrustSettings::iter::h1edabd8702ce777b + 68
    frame #7: �[33m0x0000000100dbe2b4�[0m ni3`rustls_native_certs::macos::load_native_certs::h046b3ae93eae041c + 648
    frame #8: �[33m0x0000000100db9a68�[0m ni3`rustls_native_certs::load_native_certs::ha1c97d4712715f7a + 208
    frame #9: �[33m0x0000000100d7c29c�[0m ni3`reqwest::async_impl::client::ClientBuilder::build::h24b57e6bfb97b16f + 1756
    frame #10: �[33m0x0000000100285c34�[0m ni3`nix_installer::diagnostics::DiagnosticData::send::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::hb364914f25a07dda + 3752
    frame #11: �[33m0x0000000100284c44�[0m ni3`nix_installer::diagnostics::DiagnosticData::send::_$u7b$$u7b$closure$u7d$$u7d$::hff5a58997e99305b + 1948
    frame #12: �[33m0x0000000100707ea8�[0m ni3`nix_installer::plan::InstallPlan::uninstall::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h938e184001db3b94 + 6056
    frame #13: �[33m0x00000001007065bc�[0m ni3`nix_installer::plan::InstallPlan::uninstall::_$u7b$$u7b$closure$u7d$$u7d$::hdda7160f3391c3bf + 1904
    frame #14: �[33m0x00000001003d077c�[0m ni3`_$LT$nix_installer..cli..subcommand..uninstall..Uninstall$u20$as$u20$nix_installer..cli..CommandExecute$GT$::execute::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h3b36238108552425 + 12048
    frame #15: �[33m0x00000001003cd760�[0m ni3`_$LT$nix_installer..cli..subcommand..uninstall..Uninstall$u20$as$u20$nix_installer..cli..CommandExecute$GT$::execute::_$u7b$$u7b$closure$u7d$$u7d$::he7cc9d2b5388e4aa + 1864
    frame #16: �[33m0x00000001000a69fc�[0m ni3`_$LT$core..pin..Pin$LT$P$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hb5d9d3a651fe56a0 + 80
    frame #17: �[33m0x000000010001e408�[0m ni3`_$LT$nix_installer..cli..NixInstallerCli$u20$as$u20$nix_installer..cli..CommandExecute$GT$::execute::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::ha3702df63c175800 + 2076
    frame #18: �[33m0x000000010001dae8�[0m ni3`_$LT$nix_installer..cli..NixInstallerCli$u20$as$u20$nix_installer..cli..CommandExecute$GT$::execute::_$u7b$$u7b$closure$u7d$$u7d$::ha69d753b591ad6c8 + 1828
    frame #19: �[33m0x00000001000a69fc�[0m ni3`_$LT$core..pin..Pin$LT$P$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hb5d9d3a651fe56a0 + 80
    frame #20: �[33m0x000000010000a2e0�[0m ni3`nix_installer::main::_$u7b$$u7b$closure$u7d$$u7d$::hf64e8abe886930bb + 2052
    frame #21: �[33m0x000000010000a884�[0m ni3`_$LT$tracing..instrument..Instrumented$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h1f2de4d21d0f127a + 156
    frame #22: �[33m0x000000010000b828�[0m ni3`tokio::runtime::park::CachedParkThread::block_on::_$u7b$$u7b$closure$u7d$$u7d$::hceb7e219e3e3e8fa + 56
    frame #23: �[33m0x000000010000b650�[0m ni3`tokio::runtime::park::CachedParkThread::block_on::hd1f542ec5fe59355 + 620
    frame #24: �[33m0x000000010000dbb8�[0m ni3`tokio::runtime::context::blocking::BlockingRegionGuard::block_on::h08c8b3ab3b971c54 + 140
    frame #25: �[33m0x000000010000d5c8�[0m ni3`tokio::runtime::scheduler::multi_thread::MultiThread::block_on::_$u7b$$u7b$closure$u7d$$u7d$::hd1d9237916b59e74 + 80
    frame #26: �[33m0x000000010000c3c4�[0m ni3`tokio::runtime::context::runtime::enter_runtime::h838a79fba69916b5 + 256
    frame #27: �[33m0x000000010000d4ec�[0m ni3`tokio::runtime::scheduler::multi_thread::MultiThread::block_on::hda31b1dc81786461 + 92
    frame #28: �[33m0x0000000100003c7c�[0m ni3`tokio::runtime::runtime::Runtime::block_on_inner::he9a2a95b1a4c5786 + 488
    frame #29: �[33m0x0000000100003ea8�[0m ni3`tokio::runtime::runtime::Runtime::block_on::h00b0f355995c909e + 356
    frame #30: �[33m0x00000001000072a4�[0m ni3`nix_installer::main::h160aa3f8fd00d376 + 232
    frame #31: �[33m0x000000010000e278�[0m ni3`core::ops::function::FnOnce::call_once::heb7a104604ebe398 + 20
    frame #32: �[33m0x000000010000e174�[0m ni3`std::sys::backtrace::__rust_begin_short_backtrace::hd16d71321146458a + 24
    frame #33: �[33m0x000000010000e008�[0m ni3`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h0974c10848d110d0 + 36
    frame #34: �[33m0x00000001012873b4�[0m ni3`std::rt::lang_start_internal::h9e88109c8deb8787 + 808
    frame #35: �[33m0x000000010000dfcc�[0m ni3`std::rt::lang_start::h1f83f9858182f039 + 84
    frame #36: �[33m0x000000010000732c�[0m ni3`main + 36
    frame #37: �[33m0x000000019cf0b154�[0m dyld`start + 2476

...and thread 19 is running in what I think is a Grand Central Dispatch queue to copy_certificates_from_keychain:

* thread #19, queue = �[32m'copy_certificates_from_keychain'�[0m, stop reason = �[31mEXC_BAD_ACCESS (code=10, address=0x106e34000)�[0m
  * frame #0: �[33m0x000000019cf07204�[0m dyld`dyld3::MachOFile::forEachLoadCommand(Diagnostics&, void (load_command const*, bool&) block_pointer) const + 52
    frame #1: �[33m0x000000019cf5cb6c�[0m dyld`dyld3::MachOLoaded::getSlide() const + 148
    frame #2: �[33m0x000000019cf3ced4�[0m dyld`invocation function for block in dyld4::APIs::_dyld_register_func_for_add_image(void (*)(mach_header const*, long)) + 264
    frame #3: �[33m0x000000019cf1277c�[0m dyld`dyld4::RuntimeLocks::withLoadersReadLock(void () block_pointer) + 100
    frame #4: �[33m0x000000019cf3c918�[0m dyld`dyld4::APIs::_dyld_register_func_for_add_image(void (*)(mach_header const*, long)) + 156
    frame #5: �[33m0x000000019cfe0d50�[0m libsystem_trace.dylib`_os_trace_init_slow + 616
    frame #6: �[33m0x000000019d0e53e8�[0m libdispatch.dylib`_dispatch_client_callout + 20
    frame #7: �[33m0x000000019d0e6c68�[0m libdispatch.dylib`_dispatch_once_callout + 32
    frame #8: �[33m0x000000019cfe2f30�[0m libsystem_trace.dylib`os_log_create + 856
    frame #9: �[33m0x00000001a033ba24�[0m Security`__secLogObjForCFScope_block_invoke + 40
    frame #10: �[33m0x00000001a033b9c8�[0m Security`secLogObjForCFScope + 272
    frame #11: �[33m0x00000001a033bb90�[0m Security`secLogObjForScope + 108
    frame #12: �[33m0x00000001a00d6578�[0m Security`___SecTrustSettingsUseXPCEnabled_block_invoke + 76
    frame #13: �[33m0x000000019d0e53e8�[0m libdispatch.dylib`_dispatch_client_callout + 20
    frame #14: �[33m0x000000019d0e6c68�[0m libdispatch.dylib`_dispatch_once_callout + 32
    frame #15: �[33m0x00000001a02ed7f4�[0m Security`Security::KeychainCore::TrustSettings::CreateTrustSettings(SecTrustSettingsDomain, bool, bool, Security::KeychainCore::TrustSettings*&) + 976
    frame #16: �[33m0x00000001a02decb4�[0m Security`__SecTrustSettingsCopyCertificates_block_invoke_2 + 96
    frame #17: �[33m0x000000019d0f4470�[0m libdispatch.dylib`_dispatch_block_async_invoke2 + 148
    frame #18: �[33m0x000000019d0e53e8�[0m libdispatch.dylib`_dispatch_client_callout + 20
    frame #19: �[33m0x000000019d0eca14�[0m libdispatch.dylib`_dispatch_lane_serial_drain + 748
    frame #20: �[33m0x000000019d0ed544�[0m libdispatch.dylib`_dispatch_lane_invoke + 380
    frame #21: �[33m0x000000019d0f82d0�[0m libdispatch.dylib`_dispatch_root_queue_drain_deferred_wlh + 288
    frame #22: �[33m0x000000019d0f7b44�[0m libdispatch.dylib`_dispatch_workloop_worker_thread + 404
    frame #23: �[33m0x000000019d29200c�[0m libsystem_pthread.dylib`_pthread_wqthread + 288

full session:
btall.txt

• An IPS dump from another failure: ni2-2024-11-13-102540.ips.txt
• A failed run in CI: https://github.com/DeterminateSystems/nix-installer/actions/runs/11822504396/job/32940036094?pr=1287
• Discourse thread about the update: https://discourse.nixos.org/t/the-darwin-sdks-have-been-updated/55295

[grahamc]

On Matrix, Emily theorized the new frameworks makes the Security framework available when it wasn't before, and that might be causing new unwanted behavior. Indeed, that seems to be true? So I tried adding it: #1289 ...but CI passed all the way through, suggesting that is not the issue.