Skip to content

fix(omp): set OMP_NUM_THREADS from physical cores on every platform - #805

Merged
JustVugg merged 1 commit into
JustVugg:devfrom
ThefloorMiner:fix/omp-threads-all-platforms
Aug 4, 2026
Merged

fix(omp): set OMP_NUM_THREADS from physical cores on every platform#805
JustVugg merged 1 commit into
JustVugg:devfrom
ThefloorMiner:fix/omp-threads-all-platforms

Conversation

@ThefloorMiner

Copy link
Copy Markdown

On Linux, OMP_NUM_THREADS is never set, so libgomp uses nproc — logical cores. On an SMT host that is a 2× over-subscription, and it costs 41 % of decode throughput.

The project already knows this. physical_cpu_count()'s own docstring says it:

Per-expert matmul regions are tiny and back-to-back; two SMT siblings share one AVX-512 unit and contend, so logical (SMT) counts over-subscribe and hurt throughput. We want true physical cores.

The function is correct and works on Linux. env_for() just calls it only inside if sys.platform == "win32".

And glm.c's self-exec block does not cover the gap: it sets OMP_WAIT_POLICY and GOMP_SPINCOUNT but never the thread count, and it skips itself entirely when COLI_CUDA or COLI_METAL is on — i.e. exactly the configuration where the CPU still carries every non-resident expert. The comment in env_for describes the Windows tuple as "parita' col tuning OMP self-exec di glm.c", but on this one variable the Windows path goes further than the C path rather than matching it.

Measured

GLM-5.2 744B (429 GB) on 4× RTX A6000 + EPYC 7402P (24 cores / 48 threads), CTX=32768, greedy, 64 tokens, .coli_usage restored byte-for-byte before each run. routed CPU is the PROF=1 P0-EXEC counter:

OMP_NUM_THREADS routed CPU read decode per thread
4 6.48 GB/s 1.28 tok/s 1.62 GB/s
8 12.50 1.97 1.56
16 23.97 2.63 1.49
24 (physical) 30.10 2.90 1.25
48 (logical) 17.35 2.06 0.36

Two things this says:

  • The path parallelises correctly. Near-linear from 4 to 24 threads, ~1.5 GB/s per thread. This is not a serialisation bug.
  • 48 threads read slower than 16. Per-thread throughput falls 3.5× once both siblings of each core are running the same int4 GEMV. The host's 8-channel DDR4-2400 sustains ~85 GB/s, so 17.35 GB/s at the default is 20 % of the machine.

+41 % decode on this host from the default alone.

The change

OMP_NUM_THREADS moves from the Windows-only tuple to the shared path in env_for(), still behind COLI_NO_OMP_TUNE. Verified on both platforms:

Linux (EPYC 24c/48t) macOS (M5, 10c)
default 24 (os.cpu_count() = 48) 10 (= 10)
explicit OMP_NUM_THREADS=7 7 7
COLI_NO_OMP_TUNE=1 unset unset

Windows behaviour is unchanged — same guard, same value, just hoisted.

Second change, required by the first. physical_cpu_count() gains a Darwin branch using sysctl -n hw.physicalcpu. macOS has no lscpu, so now that this patch calls the function on every platform, macOS would otherwise hit the fallback and print an over-subscription warning on hardware that has no SMT at all. With the branch it returns 10 on an M5 and stays quiet.

What I have not verified

  • One SMT host. The measurement is Zen 2 with AVX2 and no AVX-512. The docstring's reasoning is about a shared vector unit, so I would expect the effect to be at least as large on AVX-512 parts, but I have not measured one. On a non-SMT host the change is a no-op by construction (physical == logical).
  • Metal. I changed a shared path and the Metal backend also skips the C-side OMP block, so it is in scope, but my only Apple hardware is an M5 with 32 GB — too small for a run that would show anything.
  • I did not touch glm.c. A bare ./colibri invocation still gets nothing, and arguably should. That is a larger change — it needs a physical-core probe in C — and I would rather land the wrapper fix, which covers coli run, coli chat and coli web, than bundle the two.

Happy to run any additional sweep on the A6000 host; it is set up for this and .coli_usage snapshot/restore is scripted.

Only Windows ever got OMP_NUM_THREADS. glm.c's self-exec tuning sets
OMP_WAIT_POLICY and GOMP_SPINCOUNT but never the thread count, and skips
itself entirely when COLI_CUDA or COLI_METAL is on. So on Linux nothing
sets it and libgomp falls back to nproc — LOGICAL cores. On any SMT host
that is a 2x over-subscription, which is exactly what physical_cpu_count()
already warns about in its own docstring:

    two SMT siblings share one AVX-512 unit and contend, so logical (SMT)
    counts over-subscribe and hurt throughput

The function is right, it works on Linux, and env_for() called it only
under `if sys.platform == "win32"`.

Measured, GLM-5.2 744B (429 GB) on 4x RTX A6000 + EPYC 7402P, 24 cores /
48 threads, CTX=32768, .coli_usage restored byte-for-byte between runs:

    OMP_NUM_THREADS   routed CPU read   decode      per thread
       4               6.48 GB/s        1.28 tok/s   1.62 GB/s
       8              12.50             1.97         1.56
      16              23.97             2.63         1.49
      24 (physical)   30.10             2.90         1.25
      48 (logical)    17.35             2.06         0.36   <- default today

Scaling is near-linear to the physical core count, then collapses: 48
threads read slower than 16 and per-thread throughput falls 3.5x. Taking
the default from 48 to 24 is +41% decode on this host.

Both kill-switches keep working, verified: an explicit OMP_NUM_THREADS
still wins (setdefault), and COLI_NO_OMP_TUNE leaves it unset entirely.
Windows is unchanged — the value moved out of its tuple to the shared
path, same guard, same result.

Second change, needed by the first: physical_cpu_count() gains a Darwin
branch (sysctl -n hw.physicalcpu). macOS has no lscpu, so calling it there
— which this patch now does on every run — would otherwise print a
spurious over-subscription warning on hardware that has no SMT at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ThefloorMiner pushed a commit to ThefloorMiner/colibri that referenced this pull request Aug 3, 2026
…afe default

§13 is new: 2.35 -> 3.64 tok/s from three settings, two of which are
documented nowhere a user would look.

  baseline (48 threads, no CACHE_ROUTE)   2.35 tok/s
  OMP_NUM_THREADS=24                      2.70
  + CACHE_ROUTE=1 ROUTE_M=16 ROUTE_J=1    3.59   (99.1% agree, 0.9% swap)
  + CUDA_EXPERT_GB=188                    3.64

OMP_NUM_THREADS is never set on Linux (JustVugg#805): the C-side block sets the spin
policy but not the thread count, and skips itself under CUDA, so libgomp uses
nproc -- the LOGICAL count. The sweep shows why that costs 30%:

  threads   CPU expert read   decode      per thread
   4         6.48 GB/s        1.28 tok/s   1.62 GB/s
  16        23.97             2.63         1.49
  24        30.10             2.90         1.25
  48        17.35             2.06         0.36

Near-linear to physical cores, then SMT collapse. ~1.4 GB/s per Zen 2 core is
a per-core compute limit, so 24 x 1.4 = 34 GB/s is this host's ceiling and the
DDR4's ~85 GB/s is unreachable. That sets the machine's limit at ~4.5 tok/s:
at 3.64, the CPU expert read is already 200 ms of a 275 ms token.

CACHE_ROUTE (JustVugg#811) is the largest single lever and is off by default. M between
12 and 24 changes nothing -- at 45.6% residency the resident experts that
outrank the true top-8 are already inside rank 12.

§15 revised. PIN_GB=all, which this report previously recommended, is not a
safe steady state: PIN=auto seeds from .coli_usage, which grows monotonically
(18.2M -> 26.4M selections here), so the pinned set grew 184.8 -> 207.7 GB and
the engine refused to start with nothing reconfigured. Bound PIN_GB and set
COLI_USAGE_DECAY. The refusal itself is correct and is new -- before JustVugg#793 the
projection under-counted the slot width and the same config would have been
OOM-killed mid-generation instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@JustVugg
JustVugg merged commit 2e3c475 into JustVugg:dev Aug 4, 2026
13 checks passed
JustVugg added a commit that referenced this pull request Aug 5, 2026
#805 set OMP_NUM_THREADS from physical cores "on every platform" -- but it
put the setdefault in env_for(), and env_for_engine() calls that only when
arch == "glm". inkling, kimi_k3, olmoe and deepseek_v4 all took the other
branch and kept libgomp's nproc default: LOGICAL cores.

On any SMT host that is a 2x over-subscription of a memory-bound int4 GEMV
-- the collapse #718 measured at 2.3x on Zen3. Measured here on an
i7-1355U: physical_cpu_count() says 6, os.cpu_count() says 12, and
DeepSeek V4 was running the 12. The engine's own tuner prints

    [OMP] deepseek-stream: 6 thread (core fisici) invece di 12 logici

when it runs -- and it never runs on this path.

The same defect shape as the one #805 fixed, one level out: a mechanism
that reaches one engine and not its siblings.

setdefault, so an explicit OMP_NUM_THREADS still wins, and COLI_NO_OMP_TUNE
still turns the whole thing off -- both pinned by tests. OMP_NUM_THREADS=12
reproduces the old behaviour, which is also how to A/B this against the
AVX2 kernels in the previous commit rather than measuring them together.

Three tests; verified they fail without the change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants