fix(omp): set OMP_NUM_THREADS from physical cores on every platform - #805
Merged
JustVugg merged 1 commit intoAug 4, 2026
Merged
Conversation
Only Windows ever got OMP_NUM_THREADS. glm.c's self-exec tuning sets
OMP_WAIT_POLICY and GOMP_SPINCOUNT but never the thread count, and skips
itself entirely when COLI_CUDA or COLI_METAL is on. So on Linux nothing
sets it and libgomp falls back to nproc — LOGICAL cores. On any SMT host
that is a 2x over-subscription, which is exactly what physical_cpu_count()
already warns about in its own docstring:
two SMT siblings share one AVX-512 unit and contend, so logical (SMT)
counts over-subscribe and hurt throughput
The function is right, it works on Linux, and env_for() called it only
under `if sys.platform == "win32"`.
Measured, GLM-5.2 744B (429 GB) on 4x RTX A6000 + EPYC 7402P, 24 cores /
48 threads, CTX=32768, .coli_usage restored byte-for-byte between runs:
OMP_NUM_THREADS routed CPU read decode per thread
4 6.48 GB/s 1.28 tok/s 1.62 GB/s
8 12.50 1.97 1.56
16 23.97 2.63 1.49
24 (physical) 30.10 2.90 1.25
48 (logical) 17.35 2.06 0.36 <- default today
Scaling is near-linear to the physical core count, then collapses: 48
threads read slower than 16 and per-thread throughput falls 3.5x. Taking
the default from 48 to 24 is +41% decode on this host.
Both kill-switches keep working, verified: an explicit OMP_NUM_THREADS
still wins (setdefault), and COLI_NO_OMP_TUNE leaves it unset entirely.
Windows is unchanged — the value moved out of its tuple to the shared
path, same guard, same result.
Second change, needed by the first: physical_cpu_count() gains a Darwin
branch (sysctl -n hw.physicalcpu). macOS has no lscpu, so calling it there
— which this patch now does on every run — would otherwise print a
spurious over-subscription warning on hardware that has no SMT at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ThefloorMiner
pushed a commit
to ThefloorMiner/colibri
that referenced
this pull request
Aug 3, 2026
…afe default §13 is new: 2.35 -> 3.64 tok/s from three settings, two of which are documented nowhere a user would look. baseline (48 threads, no CACHE_ROUTE) 2.35 tok/s OMP_NUM_THREADS=24 2.70 + CACHE_ROUTE=1 ROUTE_M=16 ROUTE_J=1 3.59 (99.1% agree, 0.9% swap) + CUDA_EXPERT_GB=188 3.64 OMP_NUM_THREADS is never set on Linux (JustVugg#805): the C-side block sets the spin policy but not the thread count, and skips itself under CUDA, so libgomp uses nproc -- the LOGICAL count. The sweep shows why that costs 30%: threads CPU expert read decode per thread 4 6.48 GB/s 1.28 tok/s 1.62 GB/s 16 23.97 2.63 1.49 24 30.10 2.90 1.25 48 17.35 2.06 0.36 Near-linear to physical cores, then SMT collapse. ~1.4 GB/s per Zen 2 core is a per-core compute limit, so 24 x 1.4 = 34 GB/s is this host's ceiling and the DDR4's ~85 GB/s is unreachable. That sets the machine's limit at ~4.5 tok/s: at 3.64, the CPU expert read is already 200 ms of a 275 ms token. CACHE_ROUTE (JustVugg#811) is the largest single lever and is off by default. M between 12 and 24 changes nothing -- at 45.6% residency the resident experts that outrank the true top-8 are already inside rank 12. §15 revised. PIN_GB=all, which this report previously recommended, is not a safe steady state: PIN=auto seeds from .coli_usage, which grows monotonically (18.2M -> 26.4M selections here), so the pinned set grew 184.8 -> 207.7 GB and the engine refused to start with nothing reconfigured. Bound PIN_GB and set COLI_USAGE_DECAY. The refusal itself is correct and is new -- before JustVugg#793 the projection under-counted the slot width and the same config would have been OOM-killed mid-generation instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
JustVugg
added a commit
that referenced
this pull request
Aug 5, 2026
#805 set OMP_NUM_THREADS from physical cores "on every platform" -- but it put the setdefault in env_for(), and env_for_engine() calls that only when arch == "glm". inkling, kimi_k3, olmoe and deepseek_v4 all took the other branch and kept libgomp's nproc default: LOGICAL cores. On any SMT host that is a 2x over-subscription of a memory-bound int4 GEMV -- the collapse #718 measured at 2.3x on Zen3. Measured here on an i7-1355U: physical_cpu_count() says 6, os.cpu_count() says 12, and DeepSeek V4 was running the 12. The engine's own tuner prints [OMP] deepseek-stream: 6 thread (core fisici) invece di 12 logici when it runs -- and it never runs on this path. The same defect shape as the one #805 fixed, one level out: a mechanism that reaches one engine and not its siblings. setdefault, so an explicit OMP_NUM_THREADS still wins, and COLI_NO_OMP_TUNE still turns the whole thing off -- both pinned by tests. OMP_NUM_THREADS=12 reproduces the old behaviour, which is also how to A/B this against the AVX2 kernels in the previous commit rather than measuring them together. Three tests; verified they fail without the change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Linux,
OMP_NUM_THREADSis never set, so libgomp usesnproc— logical cores. On an SMT host that is a 2× over-subscription, and it costs 41 % of decode throughput.The project already knows this.
physical_cpu_count()'s own docstring says it:The function is correct and works on Linux.
env_for()just calls it only insideif sys.platform == "win32".And glm.c's self-exec block does not cover the gap: it sets
OMP_WAIT_POLICYandGOMP_SPINCOUNTbut never the thread count, and it skips itself entirely whenCOLI_CUDAorCOLI_METALis on — i.e. exactly the configuration where the CPU still carries every non-resident expert. The comment inenv_fordescribes the Windows tuple as "parita' col tuning OMP self-exec di glm.c", but on this one variable the Windows path goes further than the C path rather than matching it.Measured
GLM-5.2 744B (429 GB) on 4× RTX A6000 + EPYC 7402P (24 cores / 48 threads),
CTX=32768, greedy, 64 tokens,.coli_usagerestored byte-for-byte before each run.routed CPUis thePROF=1P0-EXECcounter:OMP_NUM_THREADSTwo things this says:
+41 % decode on this host from the default alone.
The change
OMP_NUM_THREADSmoves from the Windows-only tuple to the shared path inenv_for(), still behindCOLI_NO_OMP_TUNE. Verified on both platforms:os.cpu_count()= 48)OMP_NUM_THREADS=7COLI_NO_OMP_TUNE=1Windows behaviour is unchanged — same guard, same value, just hoisted.
Second change, required by the first.
physical_cpu_count()gains a Darwin branch usingsysctl -n hw.physicalcpu. macOS has nolscpu, so now that this patch calls the function on every platform, macOS would otherwise hit the fallback and print an over-subscription warning on hardware that has no SMT at all. With the branch it returns 10 on an M5 and stays quiet.What I have not verified
./colibriinvocation still gets nothing, and arguably should. That is a larger change — it needs a physical-core probe in C — and I would rather land the wrapper fix, which coverscoli run,coli chatandcoli web, than bundle the two.Happy to run any additional sweep on the A6000 host; it is set up for this and
.coli_usagesnapshot/restore is scripted.