Port performance fixes (upstream #497 / #464) to zion-training "Frankenstein" build

[claude-code-swe]

Port performance fixes (upstream #497 / #464) to zion-training "Frankenstein" build

This issue tracks the porting of two critical performance and safety fixes from upstream dev into our custom zion-training deployment.

Context on the "Frankenstein" Build

Our production deployment is built on upstream/dev@31526ae with custom cherry-picks (like the f6c1828 attention-only eligibility fix and METRICS parser fixes). We are intentionally not performing a full merge of the 95 upstream commits to avoid:

1. Regressions on Broadwell: A lot of the recent upstream performance work (attn: vectorize the MLA absorption helpers — +13% decode on AVX-512 hosts, byte-identical output #477, perf: SIMD MLA-absorb score + value-mix reductions (#442, +4.7x) #481, quant: int4 IDOT at S=1 on AVX-512 VNNI (g_i4s=1), like ARM SDOT (+5.5% measured) #473) is AVX-512 VNNI / AMX optimized. Our Xeon E5-2680 v4 only supports AVX2, making these changes dead weight or potential sources of compiler optimization issues.
2. Conflict Complexity: Merging the entire dev tree would introduce AMD GPU support, serve stage 2 grammar features, security hardening, and fmt=5/6 quantization, creating a massive conflict surface.

Instead, we target specific high-value, low-risk patches.

Selected Fixes for Porting

1. Upstream #497 (commit 38d1777): Fix LFRU Eviction Guard

• Why: The original LFRU eviction guard (pilot: LFRU-aware eviction guard — a speculative load never evicts a warm demand expert (#441) #474) was implemented with the comparison inverted for speculations:
  if(cs <= vs+(vs>>2)+(4u<<8)){ drop; return; }
  This demanded that a speculative load beat a just-used demand victim by 25% + 4 frequency counts. Since speculations are historically cold, the condition was almost always true, causing the cache to drop ~100% of speculative loads once full. This collapsed the LRU hit share from 27-38% down to 15%, reducing tok/s by ~15%.
• Fix: Protect the victim only when it is genuinely warm (heat >= 2 demand accesses) AND hotter than the speculation by the same margin:
  if(vs+(vs>>2)+(4u<<8) > cs){ drop; return; }
• Impact: Restores LRU churn and the 27-38% cache hit contribution, providing a +10-15% tok/s boost.

2. Upstream #464 (commit d8277d6): Reject fmt=4 without group size

• Why: Prevents silent math corruption if a fmt=4 model (grouped int4) is loaded and reaches CUDA per-row-only entry points (coli_cuda_matmul and coli_cuda_expert_mlp).
• Fix: Guard the entry points to reject fmt == 4 if gs <= 0.
• Impact: Safety guard (low risk).

Integration & Verification Plan

1. Branching: Create a new branch feat/port-lfru-guard-497 from dev.
2. Implementation:
   • Port fix(pilot): narrow LFRU eviction guard so speculations don't over-drop (#490) #497 changes in c/glm.c at pilot_realload and pilot_uring_batch.
   • Port cuda: reject fmt=4 at the per-row-only entry points (#334 follow-up to #298) #464 changes in c/backend_cuda.cu for coli_cuda_matmul and coli_cuda_expert_mlp (adapted to our signature).
3. Local Validation: Compile and run make portable and make test-c to verify correctness.
4. Canary Deploy: Deploy a test build to zion-training:/home/agent/colibri-test/.
5. Benchmarking: Run comparative A/B benchmarks to measure hit rate and tok/s delta.
6. Rollback: If testing fails, discard the colibri-test directory and rollback the branch. If successful, merge feat/port-lfru-guard-497 to dev.

[claude-code-swe]

Accidentally opened on upstream instead of fork. Re-opening on fork.