Skip to content

Six of the eight activation-width resolvers have INTERNAL linkage in qwen3_5.cpp — a new model literally cannot inherit the dtype decision, so it writes a literal (1255 of them) #426

Description

@localai-bot

The defect class is reachability, not arithmetic. #417 and #339 each report a
site that emits f32 where vLLM emits bf16. The reason those sites exist is
structural: eight functions in the tree decide an activation width, and
six of them have internal linkage — they live in the anonymous namespace of
src/vllm/model_executor/models/qwen3_5.cpp and cannot be called by any other
translation unit:

Resolver Where Linkage
Bf16GemmOutEnabled() qwen3_5.cpp:1779 internal
GdnActDType() qwen3_5.cpp:2990 internal
GdnInDType() qwen3_5.cpp:3015 internal
GdnOutDType(bool) qwen3_5.cpp:3042 internal
ResidualDType() qwen3_5.cpp:3062 internal
MergedGdnBaOutputDType(bool) qwen3_5.cpp:3094 internal
ResolveKvCacheDType() include/vllm/v1/kv_cache_dtype.h:28 shared
ResolveMambaSsmCacheDType() shared header shared

A ninth decision — the attention-block activation dtype — is computed inline as
an unnamed lambda at include/vllm/model_executor/models/dense_attn_block.h:362.
It is in a shared header but has no name, so it can only be inherited by
including the whole block.

The consequence is exactly what the two policy sections added in AGENTS.md
("Inherit its defaults, do not re-invent them") and .agents/porting.md
("Mirror the memory format, not just the math") describe: a new model author
cannot reach the decision, so they write a literal. There are 1255
DType::k{F32,BF16,F16} literals across src/vllm/model_executor/models/
(736 bf16 / 554 f32 / 6 f16), 784 of them activation-buffer allocations.
kimi_linear_device.cpp alone contains 158 of the narrow
DBuf x(d, DType::kF32, { form.

The correlation is the argument: ~30 models route through
dense_attn_block.h's single adt and have zero findings
; every model with a
dtype defect hand-rolled its own forward.

Related: include/vllm/model_executor/layers/linear.h:78 gives
LinearMethodBase::Apply an out_dtype parameter that vLLM does not have —
upstream's apply(layer, x, bias) leaves the decision to the METHOD. Its
sibling MlpGateUpMethodBase::Apply (linear.h:151) has no such parameter,
decides bf16 internally, and has had none of these bugs.

Ask

A behaviour-preserving consistency refactor: promote the width resolvers to a
shared header beside dense_attn_block.h, keeping env names and defaults
EXACTLY as they are, so a future port inherits the decision instead of
re-deriving it. Any site whose literal DISAGREES with what the resolver would
return is reported as a separate finding — repairing it is a behaviour change
and owes its own gated row (#417, #339, #401 are those rows).

This issue is the consistency half only. It does not change a single resolved
dtype.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions