glm52: calibrated NVFP4 MLA KV outer scales + serve wiring

[MadeBy561]

GLM-5.2's post-RMSNorm 512-dim kv_c latent spans ~240× across layers
(max_abs 0.02→5.2). At the writer's default outer scale of 1.0, shallow
layers quantize with E4M3-subnormal block scales; the error is amplified
downstream. s_l = max_abs(kv_c_normed)/(6·448) puts every layer's largest
block scale at the top of the E4M3 range.

Adds kv-scales/glm52-nvfp4-nf3-hybrid_mla_outer_scales_v1.json
(format nvfp4_ds_mla_outer_scale_v1; captured on
madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid, wikitext-2 2048-ctx TP4, per-rank
agreement checked, per-layer envelope with an independent capture of the same
base for shallow-layer headroom; max_abs retained for audit) and a
default-off VLLM_NVFP4_MLA_SCALES_FILE knob in serve-glm52.sh.

KLD (rtx6kpro glm52-kld-evaluation protocol, 5 fresh boots each, 4×96GB):

KV config	mean ± sd	max ctx
fp8_ds_mla	0.1263 ± 0.0030	373k
nvfp4 + scales, bf16 rope	0.1345 ± 0.0035	550k
nvfp4 + scales, fp8 rope	0.1356 ± 0.0054	600k+
nvfp4, no scales, bf16 rope	0.158	550k
nvfp4, no scales, fp8 rope	0.168	600k+

Requires a b12x build with the latent_scale identity/dynamic cache fact
(local-inference-lab/sparkinfer#52) or a one-time compile-cache clear when first enabling.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added calibrated per-layer outer-scale support for GLM-5.2 NVFP4 MLA KV caching.
  • Added a calibration artifact containing scale values and provenance metadata.
  • Added configuration to enable scales through VLLM_NVFP4_MLA_SCALES_FILE.

• Documentation

  • Added setup instructions, usage examples, evaluation results, file-format details, and cache invalidation guidance.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d38eac77-9144-4818-8f51-75dc4565978a

📥 Commits

Reviewing files that changed from the base of the PR and between 371085e and 99bf3f1.

📒 Files selected for processing (3)

• kv-scales/README.md
• kv-scales/glm52-nvfp4-nf3-hybrid_mla_outer_scales_v1.json
• serve-glm52.sh

---

📝 Walkthrough

Walkthrough

Adds GLM-5.2 NVFP4 MLA KV outer-scale calibration data, documents its format and usage, and exports a serving-script environment variable for enabling the calibrated scales.

Changes

MLA outer-scale calibration

Layer / File(s)	Summary
Calibration artifact and serving activation kv-scales/*, serve-glm52.sh	Adds per-layer nvfp4_ds_mla_outer_scale_v1 maxima and scales, documents calibration metadata, measured results, usage, and cache invalidation requirements, and exports VLLM_NVFP4_MLA_SCALES_FILE with an empty default.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• local-inference-lab/vllm#95: Adds the code path that loads and applies per-layer outer scales during NVFP4 MLA KV cache writes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: calibrated NVFP4 MLA KV outer scales and the serve script wiring.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[MadeBy561]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[malaiwah]

Minor: register VLLM_NVFP4_MLA_SCALES_FILE in envs.py to silence startup warning

When the scales file is set, vLLM logs:

WARNING [envs.py:2294] Unknown vLLM environment variable detected: VLLM_NVFP4_MLA_SCALES_FILE

This is cosmetic only — mla.py reads the var via os.getenv() directly, so it works fine. But it trips vLLM's env-var whitelist check in envs.py because the var isn't registered there.

Fix — add it to the environment_variables dict in vllm/envs.py, next to the other VLLM_B12X_MLA_CKV_GATHER_* knobs (around line 1167):

    "VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS": lambda: int(
        os.getenv("VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS", "524288")
    ),
    # Path to a nvfp4_ds_mla_outer_scale_v1 JSON file of per-layer outer
    # scales for the NVFP4 MLA KV cache writer.  Empty/unset => s_l=1.0
    # (identity, prior behavior).  Loaded by mla.py; inert unless
    # kv_cache_dtype=nvfp4_ds_mla and attention_backend=B12X_MLA_SPARSE.
    "VLLM_NVFP4_MLA_SCALES_FILE": lambda: os.getenv(
        "VLLM_NVFP4_MLA_SCALES_FILE", ""
    ),
    "VLLM_MINIMAX_M3_ENABLE_TORCH_COMPILE": lambda: bool(
        int(os.getenv("VLLM_MINIMAX_M3_ENABLE_TORCH_COMPILE", "0"))
    ),

And the matching TYPE_CHECKING entry (around line 80, after VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS):

    VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS: int = 524288
    VLLM_NVFP4_MLA_SCALES_FILE: str | None = None

The empty-string default matches mla.py's os.getenv(_NVFP4_MLA_SCALES_ENV, "") so the loader's .strip() guard works the same way.

Review and comment by madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid -- with the outer scales enabled btw 😉 for self-improvement

[voipmonitor]

I think the calibrated values should be owned by the checkpoint, not by the
vLLM source tree.

These 78 values are calibration metadata for one exact set of model weights
and one KV format (nvfp4_ds_mla). Shipping
glm52-nvfp4-nf3-hybrid_mla_outer_scales_v1.json in vLLM couples a generic
runtime to a specific model revision and makes it too easy to apply the file to
the wrong checkpoint. It also means that updating the checkpoint does not
automatically update its scales.

Suggested rework:

1. Move the calibrated values and their provenance to
   madeby561/GLM-5.2-MXFP8-NVFP4-NF3-Hybrid.
2. Prefer storing the 78 scalars directly in a versioned config.json field;
   they are small enough that this avoids sidecar discovery/download problems.
   A sidecar is also acceptable, but config.json must reference it and the
   model loader must resolve/download it relative to the selected model
   revision.
3. Keep only generic support in vLLM: parse the checkpoint metadata, validate
   the format/layer count/latent dimension/finite positive values, and expose
   the per-layer value to the existing writer/reader path.
4. Retain VLLM_NVFP4_MLA_SCALES_FILE only as an explicit experimental or
   operator override. Register it in vllm/envs.py; it currently produces an
   unknown-environment warning and can hard-fail under strict environment
   validation.
5. Log whether scales came from checkpoint metadata, an explicit override, or
   the identity default. An explicit override should take precedence.
6. Add tests for checkpoint auto-loading, override precedence, identity when
   metadata is absent, malformed/wrong-length metadata, and the uncalibrated
   draft layer remaining at identity.

The resulting ownership should be:

• checkpoint: values, calibration provenance, format version, and the model
  revision to which they apply;
• vLLM: generic schema/loader/validation and runtime application;
• environment override: debugging and experiments only.

This also needs the compile-cache correctness fix from
local-inference-lab/sparkinfer#52 before it is safe with persistent caches.
Without that fix, a kernel compiled for identity scale can be reused for a
non-identity layer and silently omit the restore multiply.

I would not merge the model-specific JSON into vLLM in its current location.