[Performance]: LRU hit share dropped 27-38% to 15% on Windows/sm_120 between 68ac9ff and 4aca059 (0.95 to 0.82 tok/s) - confounded observation, bisect offered

[mohamedmastouri2000-boop]

What happened

While running the CACHE_ROUTE ladder on my usual box (RTX 5080 sm_120 / 128 GB DDR5 / Windows 11, g64 grouped-int4, CUDA_DENSE=1 + 4 GB tier, PIPE=1 DIRECT=1, full K=8, greedy, 64-tok decode — same protocol as #467), the stock-routing baseline dropped from my long-standing ~0.95 tok/s to 0.82, and the composition of the expert hit rate changed shape:

binary	decode tok/s	hit	pin share	LRU share
dev @ 68ac9ff (many runs, #467)	0.89-0.95	84-86%	47-59%	27-38%
dev @ 4aca059 (2 runs)	0.82	74.5%	59.4%	15.0%

The pin share is stable; it's specifically the LRU contribution that collapsed (27-38% → 15%). Prefill unchanged; coherent output; no fallbacks.

Why I'm asking rather than asserting

Two changes landed between my binaries in the LRU/prefetch area — #474 (LFRU-aware pilot eviction guard) is the obvious candidate since it changes eviction decisions on exactly this path — but my observation is confounded: my .coli_usage grew ~500k selections between sessions (pin set drifted), and page-cache state differed. So this is a "does this reproduce for anyone else / does it look plausible from the diff" question, not a regression claim.

Offer

I can run a controlled bisect on this box: rebuild 68ac9ff, 1b8f307 (pre-#474) and 4aca059 (post-#474) against a frozen copy of .coli_usage, 2 runs each, same session. ~1 hour of machine time — say the word and I'll attach the table. If it's real, the interesting part is that the guard may be over-protecting demand experts into staleness on hosts where PILOT_REAL issues a lot of cross-layer loads (6-7k per 64 tokens here, per the footer).

[woolcoxm]

Root cause found — it's real, and it's #474. Full credit to @mohamedmastouri2000-boop

First: the bisect you offered is not needed. Your read of the diff was correct, and the confounders you flagged (drifting .coli_usage, page-cache state) did not cause this — the regression is deterministic from the source. Thank you for filing it as a question rather than a claim; the conservatism cost nothing here and the report was exactly right. Fix is up in #497.

What #474 actually shipped

eaa68f4 added an eviction guard to the two PILOT_REAL victim-selection sites — pilot_realload (the blocking path, the one Windows/sm_120 hits) and pilot_uring_batch (Linux io_uring). The intent, per the commit message, was sound:

a speculative load may evict a RESIDENT expert only if the predicted expert is historically hotter than the victim, by the same LFRU hysteresis tier_pick_lfru already uses (25% + 4 freq counts); otherwise it drops the speculation.

The implementation in pilot_realload:

if(g_pilot_evict_guard && m->eheat && m->elast && Sl[lru].eid>=0){
    int vid=Sl[lru].eid;
    uint64_t vs=tier_lfru_score(m->eheat[layer][vid], m->elast[layer][vid], m->eaccess_clock);
    uint64_t cs=tier_lfru_score(m->eheat[layer][eid], m->elast[layer][eid], m->eaccess_clock);
    if(cs <= vs+(vs>>2)+(4u<<8)){ drop; return; }   // <-- inverted for a speculation
}

Why this collapses the LRU share

tier_lfru_score(heat, last, clock) = (heat << 8) | recent where recent ∈ [0,255] (tier.h:30). One frequency count is worth 256 score points; recency contributes at most 255. So cs <= vs+(vs>>2)+(4u<<8) means "the speculation must beat the victim by 25% of the victim's score plus 4 full frequency counts."

That test is inverted for a speculation. A demand-loaded resident was just used — its heat is bumped every layer it touches. A speculation is, by construction, a predicted-not-yet-hot expert with little or no history. Requiring the speculation to be overwhelmingly hotter than a just-used demand victim is a condition a correct speculation almost never satisfies.

Quantified (standalone repro of the score math, 10⁶ random samples):

victim regime	guard drops the speculation
warm demand victim (heat ≥ 2, touched recently)	100%
cold victim (heat ≤ 1, stale)	100%

Once the cache is full, the guard drops ~100% of speculations. PILOT_REAL is effectively reduced to a willneed-only hinter — no real cross-layer loads land, no LRU churn happens, and the LRU contribution to the hit rate evaporates. That is precisely the shape change you measured: pin share stable (~47-59% → 59%), LRU share 27-38% → 15%, hit 84-86% → 74.5%, 0.95 → 0.82 tok/s.

This also explains why you saw no fallbacks and coherent output — the guard is cache-placement only, a dropped speculation is just demand-loaded later to the same value, so the byte-identical contract held. Only the performance shape changed, which is the sneakiest kind of regression.

The diagnostic you can run on your existing 4aca059 binary (no rebuild)

Your footer already prints the counters:

PILOT_REAL: <N> load cross-layer completati, <M> scartati (main gia' sul layer) | PILOT_K=...

Under 4aca059 I expect M ≫ N — drops dominate. Under 68ac9ff the split is healthy (drops are only the "main already owns the layer" case). That single line confirms the mechanism without a bisect. If M is large and N is small on 4aca059, this is it.

The fix (#497)

Gate the protection on the victim being genuinely warm (eheat >= 2 demand accesses) and clearly hotter than the speculation by the same 25%+4 hysteresis — i.e. test the victim against the speculation-plus-margin (protect the resident), instead of testing the speculation against victim-plus-margin (which is what shipped). This matches the commit message's stated intent ("never evict a warm demand-loaded expert") rather than the over-broad formula.

if(g_pilot_evict_guard && m->eheat && m->elast && Sl[lru].eid>=0){
    int vid=Sl[lru].eid; uint32_t vh=m->eheat[layer][vid];
    if(vh>=2){                                              // victim genuinely warm
        uint64_t vs=tier_lfru_score(vh, m->elast[layer][vid], m->eaccess_clock);
        uint64_t cs=tier_lfru_score(m->eheat[layer][eid], m->elast[layer][eid], m->eaccess_clock);
        if(vs+(vs>>2)+(4u<<8) > cs){ drop; return; }        // victim clearly warmer -> protect
    }
}

Same correction applied to pilot_uring_batch. After the fix, the score math (same 10⁶ samples):

victim regime	guard drops	guard allows (plain-LRU evict)
warm demand victim (heat ≥ 2)	100%	0%
cold victim (heat ≤ 1)	0%	100%

i.e. genuinely warm demand experts are still protected (the #441 thrash fix still holds), but cold LRU victims are again evictable — restoring the LRU churn and the 27-38% LRU hit share.

PILOT_EVICT_GUARD=0 still restores plain-LRU for a single-binary A/B. Cache-placement only → output remains byte-identical (make check 111/111, same as the #474 baseline).

Net

• You were right to point at pilot: LFRU-aware eviction guard — a speculative load never evicts a warm demand expert (#441) #474. The confounders you flagged did not contribute.
• No bisect needed; the footer-counter check above will confirm it on your existing binary in one run.
• fix(pilot): narrow LFRU eviction guard so speculations don't over-drop (#490) #497 has the corrected guard. If you can rerun your CACHE_ROUTE ladder on the PR binary, that closes the loop on the 0.95 tok/s restoration.

cc @cdhdt (author of #474) — the #441 thrash problem is real and the fix still protects against it; this just narrows the guard from "block almost everything" to "block only when the victim is genuinely warm."

[JustVugg]

Thanks for flagging it as confounded up front — that's the right instinct. A 27-38%→15% LRU hit-share drop between 68ac9ff and 4aca059 with only a 0.95→0.82 tok/s move suggests the cache accounting changed as much as the cache itself (a lot landed in that window: #298 CUDA scale rework, #421 dual-SSD, #451, the PIN/eviction paths). If you can run the git bisect you offered with a frozen .coli_usage snapshot restored before each arm (the pin set drifts otherwise, exactly as cdhdt hit on #441) and CAP_RAISE=0, that isolates it cleanly. Also worth re-checking on current dev — #474 (LFRU eviction guard) and #495 (CUDA budget clamp fix) both touch the tier that feeds LRU hit share, so the number may have moved again. Post the bisect and I'll dig in.

[JustVugg]

Resolved by #497 (merged). Your observation was not confounded at all — it was a real, deterministic regression from #474, and you caught it. Root cause: #474's eviction guard tested the speculation against the victim + a 25%+4-frequency margin, but a correct speculation is by definition colder than the resident it evicts, so once the cache filled it dropped ~100% of speculations and PILOT_REAL degraded to a willneed-only hinter — exactly the shape you measured (pin share stable, LRU share collapsing, hit down, tok/s down). #497 inverts the test to protect only genuinely-warm victims. Your own validation on the same box: 0.82 → 0.89 tok/s, LRU share 15% → 18.7%. Keeping this open only until you post the PILOT_EVICT_GUARD=0 A/B that sizes the residual ~5-10%. Thanks for reporting it carefully rather than shrugging at it.

[mohamedmastouri2000-boop]

PILOT_EVICT_GUARD A/B as requested — same binary (current dev + #497), same session, back-to-back, stock routing, usual config (CUDA_DENSE=1 CUDA_EXPERT_GB=4 CUDA_RESERVE_GB=1 PIPE=1 DIRECT=1, full K=8, greedy, 64-tok decode):

	decode tok/s	hit	LRU share
PILOT_EVICT_GUARD=0	0.96	86.6%	25.8%
guard on (#497 default)	0.89	79.4%	18.4%

So the residual is now measured: the narrowed guard costs ~8% decode and ~7 points of LRU share on this workload — and guard-off lands exactly back in the pre-#474 band (0.89–0.95 stock + #477's lift → 0.96; LRU 25.8% vs the historical 27–38%). The #497 fix removed the pathological part (~100% speculation drops); what remains is the intended protection's genuine price.

Whether that's the right default depends on the #441 thrash cost it prevents — which my box doesn't exhibit, so I can only measure the premium, not the payout. Data for the decision: if #441-class hosts see >8% from thrash protection, the default is right and this is just a documented trade; if not, the eheat>=2 threshold (or scaling it by cache pressure, as you suggested) looks worth it. Either way PILOT_EVICT_GUARD=0 is a legitimate per-box tuning knob for hosts like mine — happy to add the one-line ENVIRONMENT.md note documenting the measured trade if wanted.