DeepSeek-V4 training support

[sbhavani]

This issue tracks DeepSeek-V4 training support in Megatron Core.

DeepSeek-V4 extends DeepSeek-V3/V3.2 with hybrid compressed attention, mHC, updated routing, Muon-based training recipes, FP4 QAT, and million-token context training support.

References:

• Technical report: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf
• Hugging Face implementation PR: Add DeepSeek V4 huggingface/transformers#45616
• TileKernels mHC reference: https://github.com/deepseek-ai/TileKernels/tree/main
• Emerging Optimizers Muon reference: https://github.com/NVIDIA-NeMo/Emerging-Optimizers

Last updated: 2026-07-16

Model Architecture

• Status: Ready in Dev
• Requested Features:
  • Megatron-LM configs for DeepSeek-V4-Flash and DeepSeek-V4-Pro training
  • CSA/HCA layer schedule wired into the model spec
  • Hash routing initial MoE layer support
  • mHC and MTP support
• Overall Status:
  • Hybrid CSA/HCA Attention
  • Hash routing MoE and ClampedSwiGLU
  • mHC and MTP support

Hybrid CSA/HCA Attention

• Status: Merged in Dev
• Implementation PR:
  • [dev] [DeepSeek-v4] Part 1: Hybrid Attention with CSA and HCA #4458
  • [dev] [DeepSeek-v4] Part 4: Fusion Kernels for DSv4 Hybrid Attention #4894
• Requested Features:
  • Compressed Sparse Attention (CSA)
  • Heavily Compressed Attention (HCA)
  • Integration with Megatron attention/module specs
• Related: Feature Request: Adding NSA and other sparse attention mechanisms #4252

Hash routing MoE and ClampedSwiGLU

• Status: Merged in Dev
• Implementation PR: [dev] [DeepSeek-v4] Part 2: Hash MoE and SwiGLU clamp #4481
• Requested Features:
  • Hash routing MoE
  • ClampedSwiGLU

mHC Integration

• Status: Merged in Dev
• Implementation PRs:
  • [dev] feat(mHC): Add basic pytorch implementation of manifold hyper connection(mHC). #2943
  • [dev] fix: Support mHC with cuda graph and activation offloading #4190
  • [dev] [DeepSeek-v4] Part 3: MTP support with mHC and new mHC contract #4518
• Tracking Issues:
  • Feature Request: mHC (Manifold-Constrained Hyper-Connections) #2890
  • Design Proposal: mHC (Manifold-Constrained Hyper-Connections) #2919

Model Support on Hybrid Model

• Implementation PRs:
  • Enable Deepseek-v4 hybrid_model in dev branch Part (1/N) #5042
  • Enable Deepseek-v4 hybrid_model in dev branch Part (2/N) #5485
  • [dev] Enable DeepSeek-v4 hybrid_model Part (3/N) #5762

Sequence and Context Support

Packed Sequence

• Status: Merged in Dev
• Related PRs:
  • [Dev] add support for deepep/hybridep dispatcher under thd format training #4816
  • varlendataset for thd e2e and benchmark #4832
  • [dev] partial cuda graph support for thd format training. #4359
  • [dev] Add chunk-wise (whole-block) CUDA graph support for THD training #5258
  • [dev] [DeepSeek-v4] Packed Sequence (THD) support for DSv4 Hybrid Attention #5011
• Requested Features:
  • Packed sequence support for DSv4 Hybrid Attention
  • End-to-end packed sequence support
  • CUDA Graphs support for DSv4 THD

Long-Context Training

• Status: In Progress
• Related PRs:
  • [dev] [DeepSeek-v4] Context Parallel support #5087
• Requested Features:
  • Context parallel support for DSv4 Hybrid Attention
  • 64K to 1M context training curriculum validation

Optimizer and Training Recipe

Muon Optimizer

• Status: In Progress
• Related PR:
  • [dev] bump emerging optimizers to v0.3.0 #5320
  • Route non-Muon params through DistributedOptimizer #4771
  • Integrate LayerWiseDistributedOptimizer with DDP buffer infrastructure #4509
  • [dev] Add experimental decoupled compact LayerWise DDP layout for Muon #5388
  • [dev] Muon fp8 param gather for decoupled layout #5470
• Requested Features:
  • DeepSeek-V4 Muon/AdamW training recipe with Emerging Optimizers

FP4 QAT Recipe

• Status: Partially supported / needs DeepSeek-V4 specific validation
• Requested Features:
  • MXFP4 QAT for routed expert weights
  • Simulated FP4-to-FP8 training path with FP32 master weights
  • FP4 QK path for CSA indexer
  • BF16 index-score path for top-k selector

Performance Optimization

mHC Fusion Kernels

• Status: Merged in Dev
• Related PR:
  • [dev] mHC kernel fusion  #3828
  • [dev]: faster implementation of mHC fused kernels #4624
  • [TransformerEngine] [Common, PyTorch] Add triton mHC kernels & pytorch APIs TransformerEngine#2790
• TileLang kernels from DeepSeek https://github.com/deepseek-ai/TileKernels/tree/main

DSv4 Hybrid Attention Fusion Kernels

• Status: Merged in Dev
• The fusion kernels are released in https://github.com/NVIDIA/cudnn-frontend release 1.24.0
• Related PR:
  • [dev] [DeepSeek-v4] Part 4: Fusion Kernels for DSv4 Hybrid Attention #4894

FP8 Indexer

• Status: In Progress
• Requested Features:
  • FP8 indexer kernels
  • Integration in MCore
• Related PR:
  • DSA: Add FP8/MXFP8 and compressed Top-K indexer paths cudnn-frontend#370

Fused GroupedGEMM with ClampedSwiGLU

• Status: Merged in Dev
• Related PR: [dev] [DeepSeek-v4] Add ClampedSwiGLU to MoE mlp_op_fuser and add force balance to hash routing #5130

mHC support with EP Overlapping

• Status: In Progress
• Related PR:
  • [dev]: Fix mHC boundaries in EP overlap schedule #5471

Long-Context Optimization

• Related PR:
  • [dev] moe(perf): Refactor CP layout for better long context training performance #5664

Bug Fixes

• Related PR:
  • [dev] [fix] [DeepSeek-v4] fix dense loss and rope type in DSv4 Hybrid Attention #5018
  • Keep DeepSeek V4 CSA compressor and indexer in high precision under FP8 training #5308
  • Preserve DSA output across fused inverse RoPE #5526
  • [dev] [fix] fix MTP with contiguous CP partition mode #5706
  • Fix zero-loss indexer gradients for DSv4 CP #5809
  • [Dev] Fix distributed optimizer state save/load for mixed-dtype param groups #5835

Megatron Bridge Examples

• Status: Available
• Examples: https://github.com/NVIDIA-NeMo/Megatron-Bridge/tree/main/examples/models/deepseek_v4
• Requested Features / Coverage:
  • DeepSeek-V4-Flash proxy pretraining recipe
  • DeepSeek-V4-Pro config/provider support
  • Hugging Face ↔ Megatron checkpoint conversion
  • Inference and SFT launch examples
• Related PR:
  • [Dev] DeepSeek-V4-Flash recipe 20260610 #5266

[Minami-su]

1

[yiakwy-xpu-ml-framework-team]

NeMo AutoModel has supported SFT + FSDP2:

1. sft_fsdp : feat: DeepSeek V4 Flash support NVIDIA-NeMo/Automodel#2039
2. tilelang kernel + fsdp : feat: add optimized DeepSeek V4 kernels NVIDIA-NeMo/Automodel#2076

[sbhavani]

@YJMSTR please create a new issue with your bug

[Meirtz]

Originally reported here as a suspected fused MLA yarn-rope kernel NaN in DeepSeek-V4-Flash SFT.

Resolved — root cause was on the Megatron-Bridge side, not the kernel. See the final comment below and the fix in NVIDIA-NeMo/Megatron-Bridge#4271. (Interim debug comments removed to avoid misleading readers.)

[Meirtz]

DeepSeek-V4-Flash SFT NaN with apply_rope_fusion=True — root cause + fix (consolidated record; resolved)

• Symptom: SFT NaNs at iteration 2 ("found NaN in local forward loss calculation") with apply_rope_fusion=True; same config unfused trains clean. Reproduced on H100 and GB300, via verl and standalone Megatron-Bridge.
• Root cause (Megatron-Bridge, not this repo): the generic partial_rotary_factor → rotary_percent mapping double-applies HF's 0.125 (relative to head_dim=512) on top of qk_pos_emb_head_dim=64, which already encodes the rope split. The rope cos/sin cache shrinks to 8 dims; fused_mla_rope_inplace is then called with emb_dim=64 and reads cos/sin out of bounds (in-situ capture: healthy inputs → 1e14–1e36 garbage, nondeterministic because the OOB memory is arbitrary). The unfused path masks the same misconfig by silently rotating only 8/64 dims.
• Fix: NVIDIA-NeMo/Megatron-Bridge#4271 (rotary_percent=1.0 in the DSv4 bridge). Validated on GB300: fused-rope SFT now trains clean and matches the unfused control to 4 decimals.
• Suggestion for this repo: fused_mla_rope_inplace could assert cos.shape[-1] == emb_dim — the OOB read surfaces as nondeterministic downstream NaNs, which is expensive to localize. Happy to send that one-liner if useful.