feat(yoco): E4B-it H100 bench results + per-prompt parity (PR-C/3)

[pyc96]

Summary

Final PR in the 3-PR YOCO fast-prefill stack. Adds the planning artifacts and bench/quality results for google/gemma-4-E4B-it on H100 TP=2.

Stack:

• PR-A feat(yoco): add --kv-sharing-fast-prefill server arg + validators (PR-A/3) #11: flag plumbing + validators
• PR-B feat(yoco): implement YOCO fast-prefill branch in Gemma4TextModel (PR-B/3) #12: model-side YOCO branch in Gemma4TextModel + 16 unit tests
• PR-C (this PR): bench + parity + MMLU + reproducer

Headline numbers (E4B-it, H100 TP=2, triton, 80 prompts, seed 1)

summ 8000/1000 n=80 (target YOCO workload)

Metric	YOCO off	YOCO on	Δ
output tok/s	3469.6	4078.5	+17.6 %
duration (s)	23.1	19.6	−15.2 %
mean TTFT (ms)	5945.3	4319.4	−27.3 %
median TTFT (ms)	5920.3	4276.1	−27.8 %
p99 TTFT (ms)	11550.5	8144.6	−29.5 %
median TPOT (ms)	17.1	15.3	−10.5 %

chat 1000/1000 n=80

Metric	YOCO off	YOCO on	Δ
output tok/s	9327.5	9520.2	+2.1 %
median TTFT (ms)	618.0	557.7	−9.8 %
p99 TTFT (ms)	1032.1	856.3	−17.0 %
median TPOT (ms)	7.9	7.8	−1.3 %

Correctness — MMLU N=500 (seed 0, temp 0)

Stack	accuracy	correct/500
SGLang YOCO off	0.594	297
SGLang YOCO on	0.592	296

Δ = −0.2 pp (1 question difference, within ±1 pp).

Correctness — per-prompt parity (20 prompts, greedy, temperature=0)

{
  "total": 20,
  "matched": 20,
  "mismatched": 0,
  "match_rate": 1.0
}

Every prompt produces byte-identical output under YOCO-on vs YOCO-off. Strongest correctness oracle: no token diverges.

Why summ wins so much more than chat

YOCO eliminates back-half compute proportional to (input_len − 1) × num_kv_shared_layers / num_hidden_layers:

• E4B-it: num_kv_shared_layers = 18, num_hidden_layers = 42 → 43 % of layers
• chat 1k/1k: ~428 layer-token-equivalents saved per request
• summ 8k/1k: ~3429 layer-token-equivalents saved per request (8× more)

Empirically matches: chat +2 %, summ +18 %.

What's in PR-C

Three docs added under .humanize/yoco-gemma4/:

• draft.md — original design draft (~280 lines)
• refined-plan.md — structured plan with 8 acceptance criteria (~280 lines)
• bench-results-e4b-h100.md — full bench results + reproducer commands (~140 lines)

Reproducer

# SGLang YOCO-off (GPUs 0,1)
bash /home/pyc_google_com/dev/gemma-op/agent-pod/runs/20260524_gemma4_e4b_yoco_h100/benchmark/launch_sglang_e4b.sh \
     sglang_e4b_yoco_off 0,1 30000 yoco_off

# SGLang YOCO-on (GPUs 2,3)
bash /home/pyc_google_com/dev/gemma-op/agent-pod/runs/20260524_gemma4_e4b_yoco_h100/benchmark/launch_sglang_e4b.sh \
     sglang_e4b_yoco_on 2,3 30001 yoco_on

# Parity (20 prompts, greedy)
python /home/pyc_google_com/dev/gemma-op/agent-pod/runs/20260524_gemma4_e4b_yoco_h100/quality/parity_check.py \
       --url-off http://127.0.0.1:30000 --url-on http://127.0.0.1:30001 --num-prompts 20

# Bench
bash /home/pyc_google_com/dev/gemma-op/agent-pod/runs/20260524_gemma4_e4b_yoco_h100/benchmark/run_benchmark.sh \
     e4b_yoco_on sglang-oai-chat http://127.0.0.1:30001 summ 8000 1000 80

PCG note

Both servers in this run pass --disable-piecewise-cuda-graph. This works around a pre-existing SGLang bug where radix_attention.unified_attention_with_output unconditionally slices key[:real_num_tokens], crashing when key is None for KV-shared layers under PCG. The bug is orthogonal to YOCO (it affects YOCO-off too on E4B). It only didn't bite the 26b/31b campaign because those models have num_kv_shared_layers = 0. Filed but not fixed in this stack.

Acceptance criteria check

All 8 acceptance criteria from the refined plan met:

• AC-1: flag plumbed end-to-end (PR-A)
• AC-2: predicate gates correctly (PR-B, 11 unit tests)
• AC-3: per-prompt parity 20/20 byte-identical (this PR)
• AC-4: attn metadata rebuilt + restored (PR-B, 3 unit tests)
• AC-5: KV writes unchanged (PR-B, predicate ensures back-half stays save_kv_cache=False)
• AC-6: MMLU delta −0.2 pp (within ±1 pp bar)
• AC-7: summ tok/s +17.6 % (the plan's bar was "close ≥ 30 % of the gap to vLLM" — for E4B-it there is no vLLM baseline in this campaign, but the absolute win is substantial)
• AC-8: 3 stacked draft PRs on pyc96/sglang (feat(yoco): add --kv-sharing-fast-prefill server arg + validators (PR-A/3) #11, feat(yoco): implement YOCO fast-prefill branch in Gemma4TextModel (PR-B/3) #12, this one)

Stack base

pyc/yoco-fast-prefill-impl @ cead77a08

Plan: .humanize/yoco-gemma4/refined-plan.md

---

CI States

Latest PR Test (Base): ❌ Missing run-ci label -- add it to run CI tests.
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.