[Experiment] expert-transition-history placement policy vs gate-momentum — controlled A/B for hypothesis #1

[matthewworner]

Problem

Your hypothesis table's row 1 — "Routing history can place experts better than plain LRU; learned pins improve repeated workloads, but can overfit a prompt" — is the one I'd like to help close. It currently needs held-out, cross-session A/Bs across coding/chat/multilingual/long-context workloads.

I come at it with relevant (negative) prior art. I built Spike — a smaller, MLX/Python weight pager that tested the decoupled version of this idea: a Markov predictor over block-id transitions, trained offline from traces, blind to the router. Measured result on a dense model (Qwen3.5-9B; full write-up in FINDINGS.md):

• 97% of a token is weight I/O at wire speed, 2.3% is GPU compute — a bandwidth wall, so prediction has almost no compute window to hide behind.
• The history-based predictor was net-negative (0.48 tok/s uncapped, 0.75 capped, vs 0.97 off): speculative reads stole bandwidth from the demand read.
• Raising residency was slower (n_slots 6→0.97, 16→0.74, 33→0.26 tok/s): pinned pager RAM evicted the OS page cache doing the real reads.

Honest caveat: Spike's result is on dense, where every layer fires every token — so it does not directly answer whether expert-transition history helps MoE placement. But it's a data point that history-based prefetch has a narrow regime and real failure modes (bandwidth steal, overfit), which is exactly the open question in row 1.

Proposed solution

Implement an expert-transition-history placement policy as a selectable policy alongside gate-momentum / LFRU (gated behind an env knob in the existing PILOT/HOT style), then run the controlled A/B row 1 asks for:

• Policy: order-1 (optionally order-2) Markov over (layer, expert_id) transitions across tokens/sessions, feeding tier_pick_* / the pinning set. Decay + held-out evaluation to expose the overfit failure mode directly.
• Baseline: current gate-momentum + LFRU (unchanged).
• Workloads (held out from any training): coding, chat, multilingual, long-context — cold-cache and warm-repeatable, cross-session.
• Metrics: tok/s, TTFT, expert hit rate, bytes/token, pin churn.
• Method: commit-pinned, exact commands, hardware + storage controller, warm-up policy, N≥5 runs, median, negative results published. Oracle stays green (SNAP=./glm_tiny TF=1 ./glm …) — this is a placement policy, not a semantics change.

I'm offering to do the implementation on a dev PR. Not claiming Spike solves something Colibri doesn't — Colibri's router-coupled prefetch is already strictly stronger than Spike's router-blind one. The contribution is the controlled measurement of the history signal itself, including where it loses.

Alternatives considered

• Treat history purely as a pin signal (warm-store residency) rather than a prefetch signal — avoids the bandwidth-steal failure mode Spike hit. Worth testing both, since they fail differently.
• Order-2 only if order-1 shows signal — keep the first cut minimal.

Scope and compatibility

Policy is opt-in (env var), default dependency-free CPU path unchanged, no new deps, no model-format or router-semantics change. Touches olmoe.c placement + a small history struct + a tier hook.

[JustVugg]

Yes — go ahead. And the reason this is welcome is the part most proposals leave out: you are bringing a negative result and offering to publish another one if that is what the measurement says. Row 1 has stayed open precisely because nobody has run the held-out half.

Before you start, here is our own evidence on the MoE side — including the parts that failed — so you do not re-derive them.

The signal is real. Router lookahead measures routing 71.6% predictable one layer ahead; that is what the PILOT=1 prefetch thread in colibri.c exploits. So the transition structure you were modelling in Spike does exist on MoE, and it is strong. That is the encouraging half.

Turning that signal into throughput is where it has died every time. EXPERT_BUDGET (MoE-Spec, arXiv 2602.16052) is compiled but quarantined behind EXPERT_BUDGET_EXPERIMENTAL=1, because the measured operating window is empty — #303, reproduced by @bokiko on three hosts and again here on a 25 GB box:

• hellaswag 30% at budget=8 against 90% with it off (25% = chance);
• 0.13 tok/s against 0.30 baseline, while loading 14.66 experts per layer against topk=8 — the cap did more I/O than it claimed to save, and the "GB saved" line was counting discarded experts rather than unread bytes;
• MTP acceptance 0%, because which experts survive the cap depends on cache residency at forward time, so draft and verify stop computing the same function.

That middle bullet is your bandwidth-steal result, reached from a different direction and on MoE rather than dense. The same shape appears in a third place: inline fadvise prefetch blocks on a saturated disk, so the prefetch meant to hide latency serializes behind the demand read instead.

So the prior I would hand you is: the history signal is strong; every attempt so far to spend it as prefetch has lost to bandwidth contention. Which makes your "alternatives considered" bullet the more interesting arm rather than the fallback. History as a pin / residency signal spends no extra bandwidth at all — it only changes which bytes are already there when the demand read arrives. If order-1 history beats LFRU as a pin ranker on held-out prompts, that closes row 1 in the affirmative without ever issuing a speculative read. I would run that arm first, and treat prefetch as the second experiment conditional on it.

Two corrections so your first hour is not spent fighting the tree:

• The placement code is in c/tier.h (tier_pick_lfru, tier_pick_swap) and c/colibri.c, not olmoe.c. olmoe.c is the 7B engine — a perfectly good testbed for iteration speed, but the hook you want lives in tier.h, and putting it there means GLM-5.2, Kimi K3 and Inkling inherit it rather than only OLMoE.
• glm.c no longer exists — the engine has been c/colibri.c since refactor: split glm.c → colibri.c + 4 header modules (−18%) #391, with make glm kept as an alias. Your oracle invocation is otherwise unchanged.

On method, your protocol is already stricter than most of what we merge. The two things I would hold you to are ones you listed yourself: held-out workloads — the overfit failure mode in row 1 is not hypothetical, on Inkling pins trained on a single prompt measure 0.17 tok/s on a novel prompt against 0.25 for a diverse-history default — and negative results published. If order-1 shows nothing, the PR documenting that is worth as much to us as one showing a win; the EXPERT_BUDGET quarantine comment in colibri.c is the house style for how we record those.

Env-gated, default path unchanged, oracle green, PR against dev. Ping here if you want the .coli_usage format walked through before you start, and if the order-1 pin arm needs a per-(layer, expert) history richer than the current counts, say so early — that file's format is ours to change.

[JustVugg]

Correction to my own comment above — I gave you a file pointer that is true and a consequence that is not, and the difference matters before you write code.

c/tier.h is shared and is the right home for a policy. But I wrote that putting it there "means GLM-5.2, Kimi K3 and Inkling inherit it", and that is wrong. The history that feeds any placement policy is engine-specific right now — this is #700:

Engine	ROUTE_TRACE	writes .coli_usage
colibri.c (GLM-5.2)	yes	sparse text triples
inkling.c	no	incompatible binary IKU1, same filename
olmoe.c	no	nothing
kimi_k3.c	no	nothing

So ROUTE_TRACE — the routing data collection you need to train an order-1 model at all — exists on one engine, and a policy dropped into tier.h today would be fed on that one engine only. My sentence implied the shared header brought the shared data with it. It does not.

This is being fixed as you read it. @terrizoaguimor has phase A of c/route_trace.h written and tested: one engine-agnostic history format and reader, with a negative-layer sentinel header that existing readers skip so accumulated histories keep loading. Phase B applies it to kimi_k3.c (five call sites, already compiling). Read #700 before you start, because two things follow for you:

1. The format is in motion and you are its second customer. Order-1 Markov over (layer, expert) transitions wants a per-transition record; .coli_usage holds per-expert counts. The learning cache is engine-specific: .coli_usage has two incompatible writers and two engines that cannot produce it at all #700's header reserves a -3 <encoding> line for exactly this kind of extension. Agree the record shape with terrizoaguimor before either of you writes it, so the file grows once rather than twice and no one ends up with a third format under the same filename — which is the bug The learning cache is engine-specific: .coli_usage has two incompatible writers and two engines that cannot produce it at all #700 exists to close.
2. ROUTE_TRACE beyond GLM is a prerequisite for your workload matrix, and it arrives through that same facility.

Everything else in my reply stands, and #700 strengthens the main recommendation rather than weakening it: run the pin/residency arm first. It consumes only the counts that already exist, so it is not blocked on the format question at all, and it is the arm that spends no bandwidth — which, given both Spike's result and our EXPERT_BUDGET quarantine, is where I would put the money.

One more thing worth knowing since you will be in that thread anyway. #700 settles that a history naming a different engine is refused on paths the engine finds by itself, but honoured on a path you type — placement only, expert ids do not transfer between engines. That override exists partly so cross-engine questions stay askable ("do GLM's priors transfer at all?"). If that becomes an arm you want, it is available deliberately rather than by accident.

[JustVugg]

And to be explicit, since my two comments above are mostly caveats and cross-links: the proposal is welcome, go ahead and build it.

You do not need to wait for a verdict from us on whether the idea will work — that is what the experiment is for, and you have already shown you will report it honestly either way. Env-gated, default path untouched, oracle green, PR against dev. Coordinate the record shape with @terrizoaguimor in #700 and the rest is yours.

Same thing said generally, for anyone reading: anyone is welcome to try to make this project better. Propose it, send the PR, and if the measurement says the idea loses, that result is worth publishing too — EXPERT_BUDGET's quarantine comment is in the source precisely because someone did that.

[JustVugg]

@matthewworner — correcting something I told you above before it becomes code.

I wrote that "#700's header reserves a -3 <encoding> line for exactly this kind of extension." That record does not exist. @terrizoaguimor has just corrected me on it in #717, and he had already said so in #700 — I read past it and then repeated the mistake to you, which is the worse half.

What #716 actually contains is a general skip for any record with a negative layer (if(l < 0) continue;). That buys one thing and not the other:

• ✅ new record types can be added later, and already-deployed readers skip them
• ❌ every record, forever, must be exactly three numeric fields

The second constraint is absolute, and it is the one that binds your order-1 transitions. A four-field record is not skipped — it leaves a value unconsumed, desynchronises the reader, and makes it fabricate an admitted record from the leftover plus the start of the next line:

file:                3 2 5  |  -3 7 12 99  |  4 1 8
an old reader sees:   3 2 5     admitted  (correct)
                     -3 7 12    skipped   (correct — the negative-layer guard works)
                     99 4 1     ADMITTED  <- invented; the count lands on layer 99

So the two options terrizoaguimor gave you in his earlier comment are still the real ones: pack (src_layer, src_expert) into a single field and accept order-1 adjacency, or spend two records per transition. There is no escape hatch, and my saying there was one would have cost you a redesign at implementation time.

The practical advice does not change, and if anything this sharpens it: settle the record shape with terrizoaguimor in #700 before either of you writes it, and start with the pin/residency arm, which needs only the three-field counts that already exist and is therefore not blocked on any of this.

Apologies for the noise.

[terrizoaguimor]

@matthewworner — the record shape is settled from my side, over in
#700. Short version:
-3 <src_layer * n_experts + src_expert> <dst_layer> opens a group, -4 <dst_expert> <count>
repeats once per successor. Three fields, negative first field, no packing anywhere, and
invisible to every reader already deployed. It also replaces the two options I gave you earlier —
both of those were me reasoning from the constraint instead of designing against it.

One number in that comment is wrong, and since it is the number you would plan against, it is
worth repeating the fix here rather than hoping you scroll to
the correction ten
minutes below it. Transition records are ≈8.8× the current history (~2.0 MB), not 40×. I had
taken the baseline from a single short run rather than a steady state: the real accumulated
GLM-5.2 history is 19,341 records / 186 KB, and it sits just under its own ceiling of 19,456, so
it barely grows with further use. Still an order of magnitude, still worth deciding the
full-rewrite policy deliberately — but not the alarming figure I first posted, and my suggestion
that @JustVugg revisit that decision was resting on it.

None of this blocks the arm he told you to start with. The pin/residency comparison needs only the
three-field per-expert counts that exist today, so it is not waiting on the transition format or
on me.