Metal — growing the expert cache makes decode SLOWER

[monotophic]

[Performance]: Metal — growing the expert cache makes decode SLOWER; traced to GPU buffer-residency cost; minimal cache (--cap 1) is the fastest config on a 128 GB M5 Max

Authored in Claude Code (Opus 4.8 -> Fable 5), analysis in partnership with user id @monotophic
Suggested labels: performance, metal, macOS.

Environment

Machine	MacBook Pro M5 Max (Mac17,7), 128 GB unified, 18 cores
OS	macOS 26.5.2
Engine	main @ 72d3d37, make glm METAL=1, COLI_METAL=1, 0 local patches
Model	mateogrgic/GLM-5.2-colibri-int4-with-int8-mtp (int8 MTP heads)
SSD	internal NVMe, measured 14 GB/s @1 thread / ~20 GB/s @4–8 (F_NOCACHE random 19 MB reads)

Summary

On Metal, every expert-cache mechanism makes decode slower: raising the LRU cap, adding a
pin, or enabling AUTOPIN all reduce tok/s even as hit rate rises. The fastest configuration we
can find on this machine is a minimal cache — MTP=0 CAP_RAISE=0 AUTOPIN=0 --cap 1 — at
~2.0 tok/s, up from ~0.4 at auto defaults (5× from config alone). We ran a chain of
controlled A/Bs to find out why, falsified four mechanisms along the way, and landed on
evidence that the cost is GPU-side buffer residency / page-mapping on large, sparsely-reused
host buffers registered via newBufferWithBytesNoCopy.

The cross-platform contrast makes the point sharpest. Same engine, same model:

Box	Best published config	Hit rate	decode tok/s
DGX Spark GB10 (#161, Tier A)	large LRU + AUTOPIN + DIRECT=1	82%	2.39
M5 Max 128 GB (this report)	no cache at all (--cap 1, no pin)	2%	~2.0

The strongest memory subsystem in the tracker gets its best result by refusing to cache —
because on Metal, hits cost more than the page-cache misses they replace.

The core measurement (cap isolation; MTP=0 AUTOPIN=0 --ram 90 CAP_RAISE=0, ngen 256, temp 0)

--cap	hit	expert-matmul	GPU blocks	kernel	stall (gpu-wall − kernel)	tok/s
1	2.4%	19.2s	23,036	11.2s	5.8s	~2.0
4	12.5%	21.5s	31,741	13.2s	6.1s	1.98
16	46.6%	49.4s	37,368	12.7s	33.2s	1.52

Identical GPU arithmetic (160,106 experts on GPU, kernel time flat) — the entire regression is
stall, and it scales with how many experts are served from cache. Interleaved same-config
controls bracketed every treatment (thermal drift on this laptop is real; hit-rate/bytes are
the primary metrics, tok/s is drift-corrected color — per the methodology note in #333).

Four hypotheses tested and falsified (so nobody re-runs these dead ends)

1. Buffer register/unregister churn — dead by code inspection: slab realloc only fires on
   first fill (per-layer expert sizes are uniform); steady-state re-registration ≈ 0.
2. Hazard tracking — dead by A/B: COLI_METAL_UNTRACKED=1 across cap 1/4/16 leaves the
   slope untouched (49.4s → 48.2s at cap16; noise).
3. Per-command-buffer fixed cost / two-subset dispatch fragmentation — dead by experiment:
   an env-gated toggle that folds the resident pre-submit into the single post-load Metal command
   buffer ("CB") (one CB per layer, blocks 37,368 → 19,278 as designed) is slightly slower
   (matmul 45.9 → 54.6s at cap16). CB count is not the cost.
4. (Implicit) "a cheap fix exists" — see below; the surviving mechanism looks driver-level.

The isolating experiment

Instrumenting the three Metal MoE call sites (timers on block_begin / sync moe_block /
block_end) shows the async resident path is nearly free and the cost lives in the sync CB's
commit-to-completion wait. Then the single-CB toggle above accidentally produced the cleanest
isolation we have:

cap1 vs cap16 under the single-CB toggle = identical dispatch structure, identical 19,125
calls, identical 153,000 experts through one CB per layer. The only difference is where the
weights live: cap1 serves ~98% from the 64 constantly-reused ws[] staging slabs; cap16
serves ~45% from the ~1,200-slab LRU pool.

cap1: 1.30 ms/call · cap16: 2.86 ms/call — same work, 2.2×, purely buffer locality.

Per-resident-expert penalty ≈ 0.43 ms. (Copying hits into warm staging would cost ~0.6–0.8 ms
per 19 MB expert at measured memcpy bandwidth — routing hits through staging loses too.)

This mechanism also retro-explains COLI_MMAP=1 being catastrophic here (39× slower,
gpu-wall 1405s / kernel 9.2s): registering whole 358 GB shard files with Metal is the same
phenomenon at maximum dose — the GPU demand-faults file-backed pages the pre-touch can't keep
resident when the mapping is 3× RAM.

Questions for maintainers / Metal folks

1. Is there appetite for an MTLHeap / explicit-residency-set experiment for the expert slab
   pool? (We're happy to run controlled A/Bs on this box; the harness is built.)
2. If the cost is intrinsic to large sparsely-touched newBufferWithBytesNoCopy sets, should
   the macOS guidance/defaults change? Auto-RAM currently takes 0.88 × available (~100–113 GB
   here → 0.4 tok/s); --cap 1-style minimal-cache is 5× faster on big-RAM Macs.
3. Cross-ref telemetry: hits conflates pin-tier and ecache hits — split them so eviction-policy changes are measurable #336/telemetry: split expert hit counter into pin vs ecache (LRU) (#336) #367: our data is a concrete case for splitting pin-tier vs LRU-tier hit
   telemetry — on Metal the two tiers have opposite cost profiles.

Also confirmed on this box (supporting, not the headline)

• MTP=0 is a large win on Metal, confirming [Performance]: M5 MAX 128GB with METAL - 1.83 tok/s #87/Datapoint: M3 Ultra (Mac Studio, 256 GB, 80-core GPU) — 1.45 tok/s @ 82% hit, Metal + tuned flags #180: +79% at a cached config (1.20 → 2.15),
  +10–22% at --cap 1. Speculation was working (42% acceptance) and still lost on bytes.
• Temp-0 output is sensitive to expert-batch composition (FP accumulation order): MTP on/off
  and different cache splits can flip a near-tied token into a different-but-stable
  continuation. Worth knowing for anyone md5-verifying determinism across configs.

Reproduction

# champion config (~2.0 tok/s on this box)
MTP=0 CAP_RAISE=0 AUTOPIN=0 coli run --ram 90 --cap 1 --ngen 256 --temp 0 "<prompt>"

# the slope (each run: restore .coli_usage first; interleave controls; temp 0)
MTP=0 AUTOPIN=0 CAP_RAISE=0 coli run --ram 90 --cap {1|4|16} --ngen 256 --temp 0 "<prompt>"

Protocol details that matter: --temp 0 + fixed prompt (deterministic routing), restore
<snap>/.coli_usage before every run (every run rewrites it and changes the next AUTOPIN),
pin --ram explicitly (auto resamples MemAvailable and drifts with page-cache state), and
interleave same-config controls (compute throttles; disk-wait doesn't). Full logs, the
instrumentation diff, and per-run outputs available on request.

[JustVugg]

This is measurement-grade work — four hypotheses falsified with controlled A/Bs, interleaved same-config controls, the .coli_usage restore discipline, and a clean isolating experiment (1.30 vs 2.86 ms/call on identical arithmetic, purely by where the bytes live). The falsified-hypotheses section alone will save the next Metal investigator a week. Thank you.

Why your result makes sense (and mirrors a lesson we just learned)

The expert cache exists to save disk bytes. On your box the disk is a 14–20 GB/s internal NVMe — a miss costs almost nothing. So the cache is solving a problem you don't have, while (per your isolation) every resident-pool hit pays a ~0.43 ms Metal residency tax the constantly-reused staging slabs don't pay. Hits cost more than the misses they replace: negative-value cache, exactly as your slope shows.

This is the mirror image of the box this engine was built on (0.23 GB/s virtual disk, where a miss costs ~80 ms and the cache is everything), and it's the same lesson as the prefetcher discussion in #362: a mechanism's value is set by which resource is scarce, and the tracker now spans boxes where opposite configs win. DGX at 82% hit and you at 2% are both optimal. Config must follow the bottleneck.

Your COLI_MMAP=1 retro-explanation also coheres with what we measured on Linux (#33-era work): mmap-the-world defeats the memory budget by a different mechanism there (RSS accounting), but "registering a 358 GB mapping and letting the accelerator demand-fault it" being catastrophic is the same trap at maximum dose on both platforms.

Answers

(3) Telemetry split — already shipped. dev now has exactly what you're asking for (#336, merged yesterday): hit_pin / hit_ecache counters, surfaced as expert hit rate N% (pin X% + lru Y%) in the decode summary and (X pin + Y lru / Z load) in [PROF]. Pull dev and your cap-slope table gains the column that distinguishes the two tiers with opposite cost profiles on your platform. If the split reads wrong on Metal, that's a bug — report it.

(1) MTLHeap / explicit residency set — yes, please run it. You have the only instrumented Metal lab in the tracker and the harness is built. The experiment that would settle it: same cap-16 config, expert slab pool allocated from a single pre-resident MTLHeap (or useResidencySet:) instead of per-slab newBufferWithBytesNoCopy — if the 33 s stall collapses, the mechanism is confirmed and the fix is known; if it doesn't, the cost is intrinsic and the answer is defaults, not allocation. Either outcome is worth a comment here.

(2) Platform-aware defaults — agreed, and this should become a PR. Auto-RAM taking 0.88× of 128 GB and getting 0.4 tok/s when --cap 1 gets 2.0 is a bad default on your platform, full stop. The shape I'd take: on COLI_METAL=1, when the storage probe (doctor/iobench already measures this) reports NVMe above some threshold (say >2 GB/s), default to minimal cache and print one line saying why ([plan] Metal + fast NVMe: expert cache disabled (see #379) — override with --cap). I'd rather that logic came as a PR from your box than from here — no one maintaining this repo has Metal hardware, so you'd be writing the defaults you can actually verify, with your harness as the evidence. Happy to review line-by-line.

One small note for anyone replicating: the temp-0 near-tie flips you documented (batch composition changing FP accumulation order) match what we see on CPU too — bit-identity holds within a config, not across configs. Your framing ("different-but-stable continuation") is the right one for md5-style verification.

[monotophic]

Authored by Fable 5 in Claude Code, analysis in partnership with @monotophic

Follow-up: we built the MTLHeap experiment, and it confirms the mechanism — GPU stall −85%.

Env-gated prototype (COLI_METAL_HEAP=1, COLI_METAL_HEAP_GB=<n>, default 40): one lazily-created
MTLHeap (shared storage, untracked); expert slabs and scale slabs become heap sub-buffers
(buffer.contents replaces the malloc'd slab pointer, so the pread/compute paths are unchanged);
moe_submit issues one useHeap: instead of per-buffer useResource: for heap-backed buffers
(mixed heap/legacy sets handled via Metal's own MTLResource.heap property). NULL-fallback to the
existing path at every site; gate off ⇒ byte-identical behavior (verified against a pristine build,
same md5 output).

Same protocol as the opening post (MTP=0 AUTOPIN=0 --ram 90 CAP_RAISE=0, ngen 256, temp 0,
interleaved controls, .coli_usage restored per run), adjacent thermally-matched runs at cap 16:

cap16	expert-matmul	GPU stall (gpu-wall − kernel)	decode tok/s	output md5
stock	41.4s	25.9s	1.63	c503…
heap ON	17.8s	3.9s	1.80	c503… (identical)
heap ON, repeat	20.7s	4.8s	1.76	c503…

Heap-on cap16 matmul now sits below cap1's (~20.6s) — the per-buffer residency cost that made
cache hits expensive is gone. Prediction → targeted intervention → effect exactly where predicted,
so we're treating the buffer-residency mechanism as confirmed rather than hypothesized.

Open items:

• Heap-on adds ~+13s expert-disk wait at cap16 (absent at cap1; replicated 5×). We tested
  and eliminated two causes: heap exhaustion (fallback counter reads zero) and mutex
  serialization of sub-allocation (hoisted the alloc out of the lock — gap unchanged). Curiously
  the cost roughly mirrors the matmul savings in same-session pairs (+13s wait, −14s matmul),
  so heap-on's net decode gain at cap16 has ranged 0–10% across sessions while the GPU-stall
  elimination itself is consistent. Remaining suspicion: unified-memory ownership migration on
  CPU writes into GPU-read heap pages. Open.
• mlock() on heap-backed contents works (PIN + default MLOCK + heap: full wire, no
  failures) — so the pin tier is compatible, and pin+heap at high hit rates is now testable.
• With the fix, caching beats minimal-cache on this box for the first time. Same-session
  interleaved probe, all heap-on: cap1 → 1.655 tok/s, cap16 → 1.74 (+5%), cap32 → 1.70.
  Disk-wait finally tracks bytes (cap16 −5.5s vs cap1), and expert-matmul now decreases with
  cache size — caching behaves as designed once the per-buffer cost is gone. The optimum is
  ~cap16 here: cap32's further byte savings are eaten by page-cache displacement (RSS 53 GB).
  The win is modest on a 128 GB box because the OS page cache is an excellent fallback; on
  smaller-RAM Macs, where hit rate carries more of the load, this fix should be worth much more.
• Found in passing (pre-existing on main, not introduced by the patch): the fslab-OOM unwind in
  expert_load frees s->slab without coli_metal_unregister, leaving a stale registry entry
  over freed memory (potential GPU use-after-free if the address range is recycled). Our branch
  closes it; happy to split that one-liner into its own issue/PR if preferred.

The branch (two commits: implementation + an independent validation round) is ready to share —
happy to open a PR with the patch, the A/B protocol, and the gate-off equivalence evidence if
there's interest, or to run further controlled experiments on this box (MTLHeap sizing,
residency-set variants) if maintainers want different data first.

[claude-code-swe]

Important context for anyone applying the 'cap=1 is fastest' finding to Linux CPU-only deployments: On our zion-training VM (Xeon E5-2680 v4, QLC NVMe), cache misses cost 10-80ms each on disk. The Metal result (0.43ms/hit cache overhead > 3ms miss) does NOT apply. We're keeping our CPU-only config at AUTOPIN=1 with generous PIN_GB. The cap=1 finding is Metal-specific due to GPU buffer registration overhead.

Our full analysis: plans/colibri-performance-knowledge-base.md §12 point 4

[monotophic]

Probe-based default configs PR as requested: #386

[monotophic]

The invited work is up: #386 (platform defaults) and #392 (the residency mechanism)

[monotophic]

The invited work on this thread is now carried by PRs #386 and #426. When they merge, this is closed.