diff --git a/changelog.d/7456-mimalloc-os-tag-pre-main.md b/changelog.d/7456-mimalloc-os-tag-pre-main.md new file mode 100644 index 0000000000..edfce3b83a --- /dev/null +++ b/changelog.d/7456-mimalloc-os-tag-pre-main.md @@ -0,0 +1,9 @@ +**GC/profiling: mimalloc's VM retag now runs before `main`, so the heap really does render as `Memory Tag 240`** (fixes #7450) — #6882 set mimalloc's `os_tag` option to `VM_MEMORY_APPLICATION_SPECIFIC_1` (240) from `js_gc_init`, on the assumption that only "regions mapped before this call (early Rust startup)" would keep the default tag 100 that macOS decodes as `IOAccelerator`. mimalloc does not `mmap` per allocation: it reserves an *arena* (`mi_option_arena_reserve`, 1 GiB on 64-bit) on the first allocation and afterwards commits pages *inside* that reservation, and committing does not re-tag. The one mapping that matters was therefore made during `std`'s pre-`main` startup and the retag reached nothing — on tree.ts, ~230 MB of `IOAccelerator` against ~64 KB of `Memory Tag 240`. Labeling-only (no accounting or correctness impact), but it sends anyone following the documented profiling recipe hunting for GPU memory in a GC benchmark. + +Measured: setting the option as the *first statement of `main`* still yields a 100-tagged heap, so no placement in Rust code can fix this. The retag now runs from a `__DATA,__mod_init_func` constructor (`crates/perry-runtime/src/mimalloc_os_tag.rs`), which dyld invokes while preparing the image — before `main`, before `std`'s runtime setup, before mimalloc's arena reservation. That path must not allocate (an allocation there would reserve the arena before the tag was set, reintroducing the bug), so the `MIMALLOC_OS_TAG` opt-out is read with `libc::getenv` rather than `std::env::var_os`, which allocates an `OsString`. `js_gc_init` still calls into the module: the call selects that archive member so the constructor survives the link, and it re-applies the option idempotently. Note that the obvious way to write the keepalive — taking the address of the `#[used]` static — does *not* link: `ld` rewrites `__DATA,__mod_init_func` into a `__TEXT,__init_offsets` table of 32-bit offsets, so the static has no address left and the reference fails as `ADRP out of range ... to 0x00000000`. + +Verified end-to-end, not just in unit tests: a Perry-compiled binary under `vmmap` now reports its whole 260 MB heap (and the 768 MB reserved tail) as `Memory Tag 240` with zero `IOAccelerator`, and `otool -l` confirms the `__init_offsets` entry survived Perry's own strip/dedup link path. + +The reason this regressed invisibly is that #6882's test asserted `mi_option_get(mi_option_os_tag) == 240` — which was true the whole time it was broken. `gc/tests/os_tag.rs` now asserts the kernel's `user_tag` for a live heap address via `mach_vm_region`, i.e. exactly what `vmmap` renders; sabotaging the constructor's `link_section` fails it with `got 100` while the option-value assertion stays green. `docs/src/internals/memory-model.md` §profiling is corrected accordingly: large `IOAccelerator` regions on a current build now mean a dropped module initializer, not an expected caveat. + +Coverage caveat, stated rather than left to be discovered: the new test is macOS-only and **no required CI job runs `cargo test -p perry-runtime --lib` on a macOS runner** — `cargo-test` moved to `ubuntu-latest` in v0.5.392, where this module compiles to an empty function. So the test gates local runs and any future macOS arm, but not `main` today. Wiring it into a macOS job is a deliberate separate step (a new gate has to go green once before it can be promoted), not something this PR does silently. diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index f5b438f19e..d7e9c6303d 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -771,25 +771,16 @@ pub extern "C" fn js_gc_init() { // a second js_gc_init on another thread is harmless. #[cfg(windows)] crate::win_console::enable_vt_output(); - // #6882: mimalloc (the global allocator, #62) tags its OS mappings with - // VM tag 100, which macOS tooling — vmmap, Instruments' VM Tracker, - // `footprint` — decodes as `IOAccelerator`: the entire JS heap renders - // as GPU-driver memory (644 MB of "IOAccelerator" on an allocation-heavy - // benchmark). Retag to VM_MEMORY_APPLICATION_SPECIFIC_1 (240) so heap - // regions show up as a neutral, distinctive "Memory Tag 240" instead. - // An explicit `MIMALLOC_OS_TAG` env setting still wins — skip the - // override so profilers can keep steering the tag themselves. Regions - // mapped before this call (early Rust startup) keep tag 100; the bulk - // of the heap (arena blocks, GC metadata) maps afterwards. Idempotent, - // like the rest of this function. - #[cfg(all( - target_pointer_width = "64", - target_vendor = "apple", - feature = "alloc-mimalloc" - ))] - if std::env::var_os("MIMALLOC_OS_TAG").is_none() { - unsafe { libmimalloc_sys::mi_option_set(libmimalloc_sys::mi_option_os_tag, 240) }; - } + // #6882/#7450: macOS decodes mimalloc's default VM tag (100) as + // `IOAccelerator`, so the whole JS heap renders as GPU-driver memory in + // vmmap/Instruments/`footprint`. The retag to tag 240 that fixes this + // cannot live here — mimalloc reserves its 1 GiB arena during Rust's + // pre-`main` startup and every later allocation just commits pages inside + // that already-tagged mapping, so a retag at `js_gc_init` time reaches + // nothing (#7450). It now runs from a `__DATA,__mod_init_func` + // constructor; this call keeps that constructor in the link and re-applies + // the option idempotently. See `crate::mimalloc_os_tag`. + crate::mimalloc_os_tag::ensure_mimalloc_os_tag_applied(); crate::node_submodules::diagnostics_channel_init_main_thread(); crate::node_submodules::init_trace_events_runtime(); // #5093: force every class-field access back through the full guard call — diff --git a/crates/perry-runtime/src/gc/tests/os_tag.rs b/crates/perry-runtime/src/gc/tests/os_tag.rs index 9958539bd1..89da8f91da 100644 --- a/crates/perry-runtime/src/gc/tests/os_tag.rs +++ b/crates/perry-runtime/src/gc/tests/os_tag.rs @@ -1,24 +1,134 @@ -//! #6882: `js_gc_init` must move mimalloc's OS-mapping VM tag off 100 — -//! macOS tooling decodes tag 100 as `IOAccelerator`, so the whole JS heap -//! shows up as GPU-driver memory in vmmap/Instruments/footprint. +//! #6882/#7450: mimalloc's OS mappings must carry VM tag 240, not the default +//! 100 — macOS tooling decodes tag 100 as `IOAccelerator`, so the whole JS +//! heap shows up as GPU-driver memory in vmmap/Instruments/footprint. +//! +//! The primary assertion here is on the *mapping*, not on +//! `mi_option_get(mi_option_os_tag)`. #6882 shipped with only the +//! option-value assertion, and it passed for the entire time the feature was +//! broken: the option really was 240, it had just been set after mimalloc +//! already reserved — and thereby tagged — the arena backing the whole heap. +//! Reading the option back tests that a store landed in mimalloc's static +//! table. Reading the kernel's `user_tag` for a live heap address tests what +//! anyone following the profiling recipe actually sees. -#[cfg(all( +#![cfg(all( target_pointer_width = "64", target_vendor = "apple", feature = "alloc-mimalloc" ))] + +use crate::mimalloc_os_tag::PERRY_HEAP_VM_TAG; + +const VM_REGION_EXTENDED_INFO: libc::c_int = 13; + +/// `struct vm_region_extended_info` from ``. `user_tag` is +/// the field `vmmap` renders as the region type. +#[repr(C)] +#[derive(Clone, Copy, Default)] +struct VmRegionExtendedInfo { + protection: libc::c_int, + user_tag: libc::c_uint, + pages_resident: libc::c_uint, + pages_shared_now_private: libc::c_uint, + pages_swapped_out: libc::c_uint, + pages_dirtied: libc::c_uint, + ref_count: libc::c_uint, + shadow_depth: libc::c_ushort, + external_pager: libc::c_uchar, + share_mode: libc::c_uchar, + pages_reusable: libc::c_uint, +} + +extern "C" { + /// The calling task's port. `libc::mach_task_self()` wraps this but is + /// deprecated in favour of the `mach2` crate, which the runtime does not + /// depend on; the underlying global is stable ABI. + static mach_task_self_: libc::mach_port_t; + + fn mach_vm_region( + target_task: libc::mach_port_t, + address: *mut u64, + size: *mut u64, + flavor: libc::c_int, + info: *mut libc::c_int, + info_count: *mut libc::c_uint, + object_name: *mut libc::mach_port_t, + ) -> libc::kern_return_t; +} + +/// The kernel's VM tag for the mapping containing `addr`, or `None` if the +/// lookup failed or walked past `addr` into a later region. +fn mapping_vm_tag(addr: usize) -> Option { + let mut region_start = addr as u64; + let mut region_size: u64 = 0; + let mut info = VmRegionExtendedInfo::default(); + let mut info_count = + (std::mem::size_of::() / std::mem::size_of::()) as u32; + let mut object_name: libc::mach_port_t = 0; + // SAFETY: self-inspection of the calling task. `mach_vm_region` writes at + // most `info_count` words into `info`, and `info_count` is derived from + // `size_of::()`. + let kr = unsafe { + mach_vm_region( + mach_task_self_, + &mut region_start, + &mut region_size, + VM_REGION_EXTENDED_INFO, + (&raw mut info).cast::(), + &mut info_count, + &mut object_name, + ) + }; + if kr != 0 { + return None; + } + // `mach_vm_region` returns the first region at or above the address it is + // handed; if it skipped forward, `addr` itself is unmapped. + let contains = region_start <= addr as u64 && (addr as u64) < region_start + region_size; + contains.then_some(info.user_tag) +} + #[test] -fn js_gc_init_retags_mimalloc_os_mappings() { - // Only asserts when the profiler override isn't steering the tag — - // js_gc_init deliberately defers to an explicit MIMALLOC_OS_TAG. +fn mimalloc_heap_mappings_carry_the_perry_vm_tag() { + // The runtime defers to an operator steering the tag by hand, so the + // assertion only holds when nothing is. if std::env::var_os("MIMALLOC_OS_TAG").is_some() { return; } crate::gc::js_gc_init(); - let tag = unsafe { libmimalloc_sys::mi_option_get(libmimalloc_sys::mi_option_os_tag) }; + + // A multi-megabyte allocation through the global allocator — i.e. through + // mimalloc, i.e. backed by one of the OS mappings under test. Touched at + // both ends so the pages are real rather than a lazy reservation. + let mut heap_block = vec![0u8; 8 << 20]; + let last = heap_block.len() - 1; + heap_block[0] = 1; + heap_block[last] = 1; + + let tag = mapping_vm_tag(heap_block.as_ptr() as usize) + .expect("a live mimalloc allocation must sit inside a mapped VM region"); assert_eq!( - tag, 240, - "js_gc_init must retag mimalloc mappings to VM_MEMORY_APPLICATION_SPECIFIC_1 \ - (240); tag 100 renders the heap as IOAccelerator in macOS tools (#6882)" + libc::c_long::from(tag), + PERRY_HEAP_VM_TAG, + "mimalloc's heap mappings must carry VM tag {PERRY_HEAP_VM_TAG} \ + (VM_MEMORY_APPLICATION_SPECIFIC_1); got {tag}. Tag 100 renders the \ + heap as IOAccelerator in vmmap/Instruments (#6882), and is what you \ + get whenever the retag runs after mimalloc has already reserved its \ + arena — i.e. from anywhere other than a pre-`main` module initializer \ + (#7450). Check that the `__DATA,__mod_init_func` entry in \ + `mimalloc_os_tag` survived the link." ); } + +#[test] +fn js_gc_init_leaves_the_os_tag_option_set() { + // Strictly weaker than the mapping assertion above, and kept only as a + // localizer: if both fail, the option store itself is broken; if only the + // mapping test fails, the constructor did not run early enough (or at all). + if std::env::var_os("MIMALLOC_OS_TAG").is_some() { + return; + } + crate::gc::js_gc_init(); + let tag = unsafe { libmimalloc_sys::mi_option_get(libmimalloc_sys::mi_option_os_tag) }; + assert_eq!(tag, PERRY_HEAP_VM_TAG); +} diff --git a/crates/perry-runtime/src/lib.rs b/crates/perry-runtime/src/lib.rs index efd36597df..86740c0d07 100644 --- a/crates/perry-runtime/src/lib.rs +++ b/crates/perry-runtime/src/lib.rs @@ -87,6 +87,7 @@ pub mod macos_bundle; pub mod map; pub mod math; pub mod messaging; +pub mod mimalloc_os_tag; pub mod module_require; pub mod native_abi; pub mod native_arena; diff --git a/crates/perry-runtime/src/mimalloc_os_tag.rs b/crates/perry-runtime/src/mimalloc_os_tag.rs new file mode 100644 index 0000000000..79b5f9e2a4 --- /dev/null +++ b/crates/perry-runtime/src/mimalloc_os_tag.rs @@ -0,0 +1,111 @@ +//! macOS VM tag for mimalloc's OS mappings (#6882, repaired by #7450). +//! +//! mimalloc tags every `mmap` it makes with `mi_option_os_tag`, default 100. +//! macOS decodes tag 100 as `IOAccelerator`, so `vmmap`, Instruments' VM +//! Tracker and `footprint` render Perry's entire JS heap as GPU-driver +//! memory. #6882 retagged it to `VM_MEMORY_APPLICATION_SPECIFIC_1` (240), +//! which those tools show as the neutral `Memory Tag 240`. +//! +//! **The retag has to happen before mimalloc's first OS mapping, which means +//! before `main`.** #6882 set the option from `js_gc_init` and reasoned that +//! only "regions mapped before this call (early Rust startup)" would keep tag +//! 100. That is wrong on a per-mapping basis and catastrophically wrong in +//! aggregate: mimalloc does not `mmap` per allocation, it reserves an *arena* +//! — `mi_option_arena_reserve`, 1 GiB on 64-bit — on the very first +//! allocation, and then satisfies everything afterwards by committing pages +//! *inside* that existing reservation. Committing does not re-tag: the tag is +//! fixed by the `mmap` that created the region. So the one mapping that +//! matters is made during Rust's own startup, long before any Perry code +//! runs, and every later retag lands on an option nobody reads again. On +//! tree.ts that showed up as ~230 MB of `IOAccelerator` against ~64 KB of +//! `Memory Tag 240` — the retag covering literally the rounding error. +//! +//! Moving the call earlier inside Rust does not help; measured, setting the +//! option as the first statement of `main` still yields a 100-tagged heap, +//! because `std`'s pre-`main` runtime setup has already allocated. The only +//! placement that beats mimalloc's first mapping is a Mach-O module +//! initializer (`__DATA,__mod_init_func`), which dyld runs while preparing the +//! image — i.e. before `main`, and before any Rust allocation. +//! +//! Consequently this file runs on the pre-`main` path and **must not +//! allocate**: an allocation here would initialize mimalloc and reserve the +//! arena before we had set the tag, reintroducing the bug it exists to fix. +//! That is why the `MIMALLOC_OS_TAG` opt-out is read with `libc::getenv` +//! rather than `std::env::var_os`, which allocates an `OsString`. + +/// `VM_MEMORY_APPLICATION_SPECIFIC_1`. Chosen because macOS reserves 240-255 +/// for applications and renders them as a plain `Memory Tag `, so the heap +/// is both distinctive and obviously not a system subsystem. +#[cfg(all( + target_pointer_width = "64", + target_vendor = "apple", + feature = "alloc-mimalloc" +))] +pub const PERRY_HEAP_VM_TAG: libc::c_long = 240; + +/// Point mimalloc's `os_tag` at [`PERRY_HEAP_VM_TAG`], unless an operator is +/// already steering it with `MIMALLOC_OS_TAG` (profilers use that to isolate +/// a run; the runtime defers rather than fighting them). +/// +/// Runs before `main` from [`MOD_INIT_FUNC_ENTRY`]. Allocation-free — see the +/// module docs. Idempotent: `mi_option_set` just writes mimalloc's static +/// option table, so the second call from `js_gc_init` is a no-op write. +#[cfg(all( + target_pointer_width = "64", + target_vendor = "apple", + feature = "alloc-mimalloc" +))] +extern "C" fn apply_mimalloc_os_tag() { + // SAFETY: `getenv` is well-defined pre-`main` on Apple platforms (dyld has + // published `environ` before it runs module initializers), and takes a NUL- + // terminated string, which the C literal is. `mi_option_set` only stores + // into mimalloc's static option table — no allocation, no lock, no + // dependency on mimalloc having been initialized. + unsafe { + if libc::getenv(c"MIMALLOC_OS_TAG".as_ptr()).is_null() { + libmimalloc_sys::mi_option_set(libmimalloc_sys::mi_option_os_tag, PERRY_HEAP_VM_TAG); + } + } +} + +/// The Mach-O equivalent of `__attribute__((constructor))`: dyld walks +/// `__DATA,__mod_init_func` when the image is prepared and calls every +/// function pointer in it, before `main` and before `std`'s runtime setup — +/// the only window that precedes mimalloc's arena reservation. +#[cfg(all( + target_pointer_width = "64", + target_vendor = "apple", + feature = "alloc-mimalloc" +))] +#[used] +#[link_section = "__DATA,__mod_init_func"] +static MOD_INIT_FUNC_ENTRY: extern "C" fn() = apply_mimalloc_os_tag; + +/// Called from `js_gc_init`. Does two things, neither of them the retag that +/// actually matters — that already happened pre-`main`: +/// +/// 1. Gives the linker a reason to pull this module's object file out of +/// `libperry_runtime.a`. `#[used]` stops `-dead_strip` from dropping +/// [`MOD_INIT_FUNC_ENTRY`], but it cannot pull an otherwise unreferenced +/// archive member into the link in the first place, and at +/// `codegen-units > 1` this module is its own object. Being `#[inline(never)]` +/// and defined here, this function *is* that reason: the call from +/// `js_gc_init` selects the member, and the constructor rides along. +/// 2. Re-applies the option, covering the (currently hypothetical) case of a +/// host that ran Perry's runtime without honouring module initializers. Any +/// arena mimalloc reserves *after* this point is then tagged correctly. +/// +/// It deliberately does *not* reference `MOD_INIT_FUNC_ENTRY` from code, which +/// is the obvious way to write step 1 and does not link: `ld` rewrites +/// `__DATA,__mod_init_func` into a `__TEXT,__init_offsets` table of 32-bit +/// offsets, so the static has no address left to take and the reference fails +/// as `ADRP out of range ... to 0x00000000`. +#[inline(never)] +pub fn ensure_mimalloc_os_tag_applied() { + #[cfg(all( + target_pointer_width = "64", + target_vendor = "apple", + feature = "alloc-mimalloc" + ))] + apply_mimalloc_os_tag(); +} diff --git a/docs/src/internals/memory-model.md b/docs/src/internals/memory-model.md index e0398f84f5..31a84e8589 100644 --- a/docs/src/internals/memory-model.md +++ b/docs/src/internals/memory-model.md @@ -178,13 +178,28 @@ Two things to know before reading `vmmap`, Instruments' VM Tracker, or zones. - **Those regions used to render as `IOAccelerator`** — i.e. GPU driver memory — because mimalloc tags its mappings with VM tag 100, which macOS - tooling decodes as `IOAccelerator`. Since #6882 the runtime retags them to - `VM_MEMORY_APPLICATION_SPECIFIC_1` (240) during `js_gc_init`, so the JS - heap shows up as **`Memory Tag 240`**. If you see large `IOAccelerator` - regions in a Perry process, you are either on a pre-#6882 build or looking - at the few pages mapped before `js_gc_init` ran; there is no GPU memory - involved. Set `MIMALLOC_OS_TAG=` to steer the tag yourself — the - runtime's retag defers to an explicit env setting. + tooling decodes as `IOAccelerator`. The runtime retags them to + `VM_MEMORY_APPLICATION_SPECIFIC_1` (240), so the JS heap shows up as + **`Memory Tag 240`** and there is no GPU memory involved. Set + `MIMALLOC_OS_TAG=` to steer the tag yourself — the runtime defers to an + explicit env setting. + + The retag runs from a `__DATA,__mod_init_func` constructor + (`perry-runtime/src/mimalloc_os_tag.rs`), *not* from `js_gc_init`, and that + placement is load-bearing: mimalloc reserves a 1 GiB arena on its first + allocation — during `std`'s pre-`main` startup — and every later allocation + just commits pages inside that already-tagged region, so setting the option + from any Rust code, `main` included, retags nothing that matters. #6882 set + it from `js_gc_init` and was therefore inert for the whole heap until #7450 + moved it pre-`main`. On a build between those two, expect ~all of the heap + as `IOAccelerator` and a token `Memory Tag 240` region. + + So: large `IOAccelerator` regions on a current build are a *bug*, not a + documentation caveat — most likely the module initializer was dropped from + the link. `crates/perry-runtime/src/gc/tests/os_tag.rs` asserts the real + thing (the kernel's `user_tag` for a live heap address, via + `mach_vm_region`); note that `mi_option_get(mi_option_os_tag)` reports 240 + in the broken case too, so it is not a diagnosis. Also note that mimalloc purges freed memory with `MADV_FREE`-style advice: macOS keeps such pages counted in RSS and `phys_footprint` until memory