[Bug]: OpenMP spin-wait tuning is inert on macOS — and applying it costs 2.2x decode on a 32 GB host

[fredchu]

Summary

The OpenMP hot-thread tuning block in main() never reaches the OpenMP runtime on macOS, because the self-re-exec that makes it effective is compiled only for Linux and FreeBSD. On this host that turns out to be load-bearing in the good direction: applying the same settings externally makes decode 2.2x slower. So the report is less "please fix the missing branch" and more "please don't fix it without gating it, and here is the measurement".

Two separate items, both about how the OpenMP team is configured on macOS.

Commit or version

2d623816eecbf740b1fc06eb422974bda0af8931 (main, 2026-07-28)

Environment

macOS 26.5.2 (Darwin 25.5.0), Apple M1 Max (8 performance + 2 efficiency cores), 32 GB unified memory, internal Apple NVMe / APFS, Apple clang 17.0.0, Homebrew libomp, make colibri METAL=1. Model: mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp (429.3 GB, int8 MTP head). Full benchmark context in the companion performance report.

Reproduction steps

cd colibri/c && make colibri METAL=1

P="Explain why a Mixture-of-Experts model needs less RAM than its parameter count."

# A — as shipped
RAM_GB=24 COLI_METAL=1 ./coli run --model <model> --temp 0 --ngen 32 "$P"

# B — the same four variables main() tries to set, exported before launch
RAM_GB=24 COLI_METAL=1 \
OMP_WAIT_POLICY=active KMP_BLOCKTIME=200 OMP_DYNAMIC=FALSE GOMP_SPINCOUNT=200000 \
  ./coli run --model <model> --temp 0 --ngen 32 "$P"

# then one variable at a time

Expected behavior

Either the tuning applies on macOS the way it does on Linux, or it is explicitly not applied because it was measured not to help there.

Actual behavior and logs

=== Item 1: the tuning is inert on macOS, and that is currently a good thing ===

decode 32 tokens, prompt 24 tokens, greedy, RAM_GB=24, COLI_METAL=1
expert hit rate held at 9.9-10.1% across every run below, so cache state is not the variable

  A  as shipped                          218.55s / 219.39s   (mean 218.97s)
  B  four tuning vars exported           481.20s / 490.72s   (mean 485.96s)   +122%
  C  OMP_NUM_THREADS=8                   209.57s / 210.01s   (mean 209.79s)   -4.2%
  D  OMP_NUM_THREADS=8 + tuning vars     233.00s / 234.51s   (mean 233.76s)   +6.8%

A != B is itself the evidence that the in-process setenv() is not reaching the
runtime: if it were, exporting the same values would be a no-op.

Single-variable isolation (n=1 each, same conditions):

  baseline (nothing set)                 218.97s      --
  OMP_WAIT_POLICY=active                 485.76s     +122%
  KMP_BLOCKTIME=200                      471.40s     +115%
  OMP_DYNAMIC=FALSE                      220.78s      +0.8%  (noise)
  GOMP_SPINCOUNT=200000                  217.39s      -0.7%  (noise; libomp
                                                      ignores GOMP_*, as expected)

Either spin-wait knob alone reproduces the full regression. The two non-spin
knobs are inert. Same-config repeats differ by <=2%, so these are real.

Note the interaction with team size: the tuning costs +122% at 10 threads but
only +6.8% at 8 threads (B vs D). Leaving 2 cores unclaimed by the OMP team
turns a catastrophe into a minor loss.

=== Item 2: OpenMP team size includes the efficiency cores ===

  [plan] warning: lscpu core probe failed: [Errno 2] No such file or directory: 'lscpu'
  [plan] warning: physical-core probes unavailable; using 10 logical CPUs
                  (SMT may over-subscribe). Set OMP_NUM_THREADS to physical
                  cores if slow.

  OMP_NUM_THREADS unset (10 threads)     218.97s
  OMP_NUM_THREADS=8                      209.79s     -4.2%

Item 1 — why it is inert, and why not to "fix" it naively

c/colibri.c seeds the tuning with setenv(..., 0) and then, per its own comment, relies on a self-re-exec because libgomp reads the environment in its constructor, before main():

if(!getenv("COLI_OMP_TUNED") && !getenv("COLI_NO_OMP_TUNE") &&
   !getenv("COLI_CUDA") && !getenv("COLI_METAL")){
    setenv("OMP_WAIT_POLICY","active",0);
    setenv("GOMP_SPINCOUNT","200000",0);
    setenv("KMP_BLOCKTIME","200",0);
    setenv("OMP_PROC_BIND","close",0);
    setenv("OMP_DYNAMIC","FALSE",0);
    setenv("COLI_OMP_TUNED","1",1);
#ifdef __linux__
    ... execv("/proc/self/exe", argv);
#endif
#ifdef __FreeBSD__
    ... execv("/proc/curproc/file", argv);
#endif
}

There is no Darwin branch, so on macOS the setenv calls land and nothing re-reads them. I initially assumed LLVM libomp's lazy initialization (first parallel region, after main()) might pick them up anyway, which would have made this a non-issue — the A vs B measurement above rules that out.

The reason this matters: adding _NSGetExecutablePath() + execv to close the platform gap looks like an obvious cleanup, and on this host it would cost 2.2x decode throughput. The mechanism is plausible enough to expect on any low-residency host: at ~10% residency the engine leans on the bounded async I/O pool (PIPE_WORKERS, default 8) to hide ~86% of disk service time, and a spin-waiting OMP team sized to every core starves those I/O threads. The KMP_BLOCKTIME=200 mitigation added for #341 is not short enough here — with one small parallel region per expert, 200 ms of blocktime means the team effectively never sleeps.

Suggestions, in order of how confident I am:

1. If a Darwin re-exec is ever added, exclude both spin-wait settings there, or gate the whole block on measured expert residency — the settings are tuned for hosts where experts are resident and compute dominates, which is the opposite of this regime.
2. Consider documenting OMP_WAIT_POLICY / KMP_BLOCKTIME in docs/tuning.md as knobs that can be strongly negative on disk-streaming hosts. Right now a reader who finds them in the source is likely to try exporting them.
3. Separate papercut in the same guard: it tests getenv("COLI_CUDA") / getenv("COLI_METAL") for presence, not value, so COLI_CUDA=0 and COLI_METAL=0 disable the tuning too. That has a reachable consequence beyond users hand-setting them — coli itself sets e["COLI_CUDA"]="0" at c/coli:239 for --auto-tier --gpu none, so on a CPU-only Linux host that documented invocation appears to skip the tuning that a bare coli run would get. I read that from the source and did not measure it (no Linux host here). atoi(getenv(...)) would cover both cases.

Item 2 — sizing the team from performance cores on Apple Silicon

physical_cpu_count() in c/resource_plan.py has a Windows branch and an lscpu branch; macOS falls through both to os.cpu_count() = 10, and warns about SMT. Apple Silicon has no SMT, so the warning's framing does not apply — but the number is still wrong for this purpose, because 2 of those 10 cores are efficiency cores. With OMP_DYNAMIC=FALSE and a barrier per expert matmul, the slowest thread paces the team.

Worth noting that a "physical cores" probe would not help here: hw.physicalcpu is also 10 on this chip. The value you want is the performance-core count:

sysctl -n hw.perflevel0.logicalcpu   # 8 on M1 Max
sysctl -n hw.perflevel1.logicalcpu   # 2 (efficiency)
sysctl -n hw.physicalcpu             # 10  <-- not the number you want

hw.perflevel* exists on all Apple Silicon; on Intel Macs it is absent, so falling back to hw.physicalcpu there is correct. Measured gain from getting this right: -4.2% decode, -6% prefill on this host, for free.

Happy to test patches for either item on this hardware.

[fredchu]

Full benchmark context for this host — configuration matrix, Metal residency analysis, and the disk-traffic verification — is in #706.

[JustVugg]

All three items check out in the source. Thank you for measuring instead of assuming — especially on item 1, where the obvious-looking cleanup is the wrong move and only a number could show it.

Item 1 — the missing Darwin branch is load-bearing, and your measurement turns it from an oversight into a decision

Confirmed: the guard is at c/colibri.c:7583 and the re-exec exists only under #ifdef __linux__ / #ifdef __FreeBSD__. There is no Darwin path, so on macOS the setenv calls land in an environment that nobody re-reads. Your A/B rules out the lazy-init escape hatch, which was the one thing that could have made this benign.

Your 2.2x is the first quantified macOS number, and it points the same way as everything else that has been added to that guard, one platform at a time:

• [Bug]: glm stays at 3000% CPU after the model finished the answer. #341 (FreeBSD) — under LLVM libomp, OMP_WAIT_POLICY=active implies KMP_BLOCKTIME=infinite; a team parked on stdin was measured at 3000% CPU. That is the only reason KMP_BLOCKTIME=200 is in the block.
• docs: Metal backend (#72) — M5 Max performance report, OMP-spin regression + fix #116 (Metal) — active spinning reported at ~39%, which is why COLI_METAL is in the guard.
• [Bug]: ~3x throughput regression on x86+CUDA since 92011de — OMP hot-spin tuning vs new threads (I4_ACC512 exonerated; COLI_NO_OMP_TUNE=1 recovers and then some) #159 / OMP hot-thread tuning is skipped whenever COLI_CUDA is set — but most experts still run on CPU when the model exceeds VRAM #669 (x86 + CUDA) — a ~3x regression reported, which is why COLI_CUDA is there too.

Every one of those says the same thing: where the OMP team is not the thing doing the work, keeping it hot costs more than it saves. Your host is the cleanest instance yet. At ~10% residency the token is PIPE_WORKERS waiting on APFS, and a spinning team sized to all 10 cores contends for exactly the cores that would otherwise service the read. Your mechanism reading matches ours, and your point about KMP_BLOCKTIME=200 being far longer than a per-expert region is right — 200 ms of blocktime against one small parallel region per expert means the team effectively never sleeps.

So we will do what you ask rather than the naive cleanup: make the exclusion explicit at the #ifdef — a comment stating macOS is deliberately not given a re-exec branch, citing this issue — plus your suggestion (2), a docs/tuning.md note that OMP_WAIT_POLICY / KMP_BLOCKTIME can be strongly negative on disk-streaming hosts. Someone who finds those in the source and tries exporting them should meet the warning first.

Item 1, sub-point 3 — real bug, and reachable exactly as you describe

c/coli:273 does e["COLI_CUDA"]="0", and the engine tests presence, so on a CPU-only Linux host that documented invocation silently skips tuning a bare run would get. One distinction worth making before patching, because the guard currently treats four variables identically when only two are the same kind of thing:

• COLI_NO_OMP_TUNE and COLI_OMP_TUNED are kill-switches, and presence-based is deliberate there — c/coli documents it in a comment ("impostarla a qualsiasi valore, anche 0, disattiva"). Those should stay getenv() != NULL.
• COLI_CUDA and COLI_METAL are state, with a meaningful 0. Those want the value.

So atoi() on the last two only, not all four. Good catch, and thank you for saying plainly that you read it out of the source and could not measure it.

Item 2 — accepted, and hw.perflevel0 is the part we would have gotten wrong

physical_cpu_count() at c/resource_plan.py:245 has a win32 branch and an lscpu branch and nothing else; macOS falls through to os.cpu_count() with an SMT warning that does not apply to Apple Silicon. The useful half of your report is the observation that hw.physicalcpu is also 10 on this chip — a patch that just "adds a physical-core probe for macOS" would have fixed the warning text and left the number wrong. With OMP_DYNAMIC=FALSE and a barrier per expert matmul the two E-cores pace the team, so hw.perflevel0.logicalcpu is the value, falling back to hw.physicalcpu on Intel where perflevel* is absent. -4.2% decode and -6% prefill for free is worth having.

Patches

You offered to test — better still, take them, as PRs against dev:

1. c/resource_plan.py — Darwin branch: hw.perflevel0.logicalcpu → hw.physicalcpu → os.cpu_count(), and drop the SMT wording on that path.
2. c/colibri.c — atoi() for COLI_CUDA/COLI_METAL only, plus the explicit macOS-exclusion comment at the #ifdef.
3. docs/tuning.md — the negative-knob note, citing this issue.

Separately or together, whichever suits. If you would rather only measure, say so and we will write them for you to A/B on the M1 Max. #706 is the wider context for this host and gets its own reply.

[JustVugg]

Independent confirmation of your measurement, from different hardware and a different engine.

@rgbkrk just landed #750, which among other things excludes Apple from the OpenMP spin block in inkling.c:

-#ifndef COLI_CUDA
+#if !defined(COLI_CUDA) && !defined(__APPLE__)

with this reasoning, arrived at without reference to your report:

the active-spin team steals the shared SoC power budget … measured strictly worse on Inkling-Small even for pure-CPU decode — 0.50 vs 0.84 tok/s, and 2x the prefill

You measured 2.2× slower decode on an M1 Max (32 GB) running GLM-5.2. He measured ~1.7× slower decode and 2× prefill on a 128 GB Mac running Inkling. Different machine, different memory size, different engine, different residency regime — same direction, comparable magnitude.

That matters because your report had one honest weakness, which you named yourself: it was one host. Two independent measurements agreeing turns "a tuning preference on this Mac" into a property of the platform, and it is now the strongest evidence in this tracker about that block.

His mechanism is also a better explanation than mine. I described it as the spinning team starving the PIPE_WORKERS I/O pool, which fits your ~10% residency host but not his — he is at 128 GB with a warm page cache and still measured it worse. Shared SoC power budget covers both: on Apple Silicon the cores and everything else draw from one envelope, so a team burning cycles in a spin loop takes power away from whatever is doing the real work, whether that is I/O or the GPU. It also explains #116's 39% Metal regression without needing a separate story.

What this changes for the three items in your report:

1. The Darwin exclusion is now a measured decision rather than an accident. inkling.c has it explicitly; colibri.c and kimi_k3.c still need the same treatment, and colibri.c wants care because 14 open PRs touch that file.
2. The docs/tuning.md note — still owed, and now with two datapoints to cite instead of one.
3. COLI_CUDA presence-vs-value — fixed in fix(omp): COLI_CUDA/COLI_METAL are state, not kill-switches — test the value (#707) #739, shipped in v1.4.0. Measured before and after: with the variable set to zero it went from silently skipping the tuning to running it. That was the half you said you could not test without a Linux host; consider it tested.

Thank you for filing it the way you did. "Please do not fix this without gating it, and here is the measurement" was the right framing, and it is the reason the exclusion landed as a deliberate line with a comment rather than as somebody quietly adding a Darwin execv and making every Mac slower.