mindspeed-MM qwen3.5-4b lora 微调报错：
错误：
核心问题不是前面的 warning，而是 LoRA 权重校验阶段触发了 HCCL 不支持的通信算子。
真正报错：RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
触发位置：mindspeed_mm/fsdp/utils/lora_weight_manager.py 的 verify_lora_weights()
它对 DTensor 调用了 full_tensor()，底层会走 all-gather。
在你当前 NPU + HCCL 环境下，这条 all-gather 路径（allgather_into_tensor_coalesced）不支持，所以多卡都在初始化模型后一起崩了。
最后的 awk syntax error 是连带现象：训练提前失败后，统计脚本拿不到迭代时间变量，才报这个错，不是主因。
前面这些基本都不是致命：
OMP_NUM_THREADS=1：性能提示
pkg_resources is deprecated：依赖告警
fast path is not available：会回退到 torch 实现，可跑但可能慢
环境：npu910b 8X32GB
npu-dirver 25.5.1
can 8.5.0
docker-image: swr.cn-south-1.myhuaweicloud.com/ascendhub--mindspeed-mm:2025.rc4-a2-arm
model: qwen3.5-4b
python 环境：
accelerate 1.2.0
gpytorch 1.14.3
mindspeed 0.12.1 /data/work/git/MindSpeed
mindspeed-mm 0.1 /data/work/git/MindSpeed-MM
numpy 1.26.0
torch 2.7.1
torch_npu 2.7.1.post2
torchaudio 2.7.1
torchdata 0.11.0
torchvision 0.22.1
transformers 5.2.0
verl_npu 0.0.1 /data/work/git/MindSpeed-MM/verl_plugin
任务：mindspeed-MM qwen3.5-4b Lora 微调
使用 examples/qwen3_5/finetune_qwen3_5_4B.sh
examples/qwen3_5/qwen3_5_4B_config.yaml 内容
并行策略
parallel:
fully_shard_parallel_size: auto
fsdp_plan:
apply_modules: # 如果要开prefetch的话，请不要随意修改apply_modules的顺序
- model.visual
- model.visual.blocks.{}
- model.language_model
- model.language_model.embed_tokens
- model.language_model.layers.{}
- lm_head
param_dtype: bf16
reduce_dtype: fp32
recompute: true
recompute_plan:
apply_modules:
- model.language_model.layers.{*}
ulysses_parallel_size: 1 # 开启 ulysses-cp 时, 请将 model 的 attn_implementation 设置为 flash_attention_2
数据相关配置
data:
dataset_param:
dataset_type: huggingface
#数据集属性
attr:
images: images
messages: messages
role_tag: role
content_tag: content
user_tag: user
assistant_tag: assistant
# 数据预处理
preprocess_parameters:
model_name_or_path: &HF_MODEL_LOAD_PATH ./ckpt/hf_path/Qwen35-xxB # 替换为原始hf权重
use_fast_tokenizer: true
split_special_tokens: false
image_max_pixels: 262144
image_min_pixels: 1024
video_max_pixels: 16384
video_min_pixels: 0
video_fps: 2.0
video_maxlen: 64
basic_parameters:
cutoff_len: 1024
template: qwen3_vl_nothink
enable_thinking: false
train_on_prompt: false
mask_history: false
dataset_dir: ./data
dataset: &DATASET_PATH ./data/mllm_format_llava_instruct_data.json
cache_dir: ./cache_dir/
overwrite_cache: false
preprocessing_batch_size: 1000
preprocessing_num_workers: 16
max_samples: null
数据加载
dataloader_param:
pin_memory: true
shuffle: false
dataloader_mode: sampler
drop_last: true
sampler_type: BaseRandomBatchSampler
num_workers: 8
collate_param:
model_name: qwen3vl
ignore_pad_token_for_loss: true
模型配置
model:
model_id: qwen3_5
model_name_or_path: *HF_MODEL_LOAD_PATH
trust_remote_code: true
attn_implementation: sdpa
freeze:
- model.visual
loss 配置
loss_cfg:
loss_type: default # If you want raw loss in model, loss_type can be set to "raw".
router_aux_loss_coef: 0.0
chunkloss配置
enable_chunk_loss: true
chunkloss_plan:
apply_module: lm_head
chunk_size: 1024
activation offload 配置
enable_activation_offload: false
activation_offload_plan:
apply_modules:
- model.visual.blocks.{}
- model.language_model.layers.{}
融合算子配置
use_triton_gdn: true
训练配置
training:
micro_batch_size: 1
gradient_accumulation_steps: 1
seed: 42
lr: 1.0e-4
lr_decay_style: cosine
lr_warmup_ratio: 0.1
weight_decay: 0
train_iters: 10000
clip_grad: 0.0
init_model_with_meta_device: true
optimizer: adamw
adam_fused: true
save_interval: 10000
no_load_optim: true # Do not load optimizer state; remove if loading is needed.
no_load_rng: true # Do not load RNG state; remove if loading is needed.
no_save_optim: true # Do not save optimizer state; remove if saving is needed.
no_save_rng: true # Do not save RNG state; remove if saving is needed.
LoRA：仅训练低秩适配器，需安装 peft；续训可设置 pretrained_lora_path
lora:
enable: true
rank: 8
alpha: 16
dropout: 0.0
init_lora_weights: true
pretrained_lora_path: null
target_modules:
- model.language_model.layers.{}.self_attn.q_proj
- model.language_model.layers.{}.self_attn.k_proj
- model.language_model.layers.{}.self_attn.v_proj
- model.language_model.layers.{}.self_attn.o_proj
- model.language_model.layers.{}.mlp.gate_proj
- model.language_model.layers.{}.mlp.up_proj
- model.language_model.layers.{*}.mlp.down_proj
load: ./ckpt/hf_path/Qwen35-xxB-dcp # 替换为转换后的dcp权重
save: ./save_path
use_deter_comp: false
plugin:
- mindspeed_mm/fsdp/models/qwen3_5
- mindspeed_mm/fsdp/data/datasets/huggingface
工具配置
tools:
profile:
enable: false
profile_type: static
ranks: [0]
static_param:
level: level1
with_stack: false
with_memory: false
record_shapes: false
with_cpu: true
save_path: ./profiling
start_step: 10
end_step: 11
data_simplification: false
aic_metrics_type: PipeUtilization
memory_profile:
enable: false
start_step: 1
end_step: 2
save_path: ./memory_snapshot
dump_ranks: [0]
stacks: all
max_entries: null
mem_info: false
报错信息：
省略了一些日志
[rank2]: Traceback (most recent call last):
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 335, in
[rank2]: trainer = Trainer(args=args)
[rank2]: ^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 71, in init
[rank2]: self.model = self.get_model(model_provider)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 179, in get_model
[rank2]: self.lora_weight_manager.verify_lora_weights()
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 371, in verify_lora_weights
[rank2]: param_data = self._gather_dtensor(param.data) if self._is_dtensor(param.data) else param.data
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 84, in _gather_dtensor
[rank2]: full_tensor = param_data.full_tensor()
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 570, in full_tensor
[rank2]: redist_res = self.redistribute(
[rank2]: ^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 542, in redistribute
[rank2]: return Redistribute.apply(self, device_mesh, placements, async_op)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
[rank2]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 306, in forward
[rank2]: output = redistribute_local_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 213, in redistribute_local_tensor
[rank2]: new_local_tensor = current_placement._to_replicate_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 260, in _to_replicate_tensor
[rank2]: result = funcol.all_gather_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/_functional_collectives.py", line 205, in all_gather_tensor
[rank2]: tensor = torch.ops._c10d_functional.all_gather_into_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
[rank2]: return self._op(*args, (kwargs or {}))
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
[rank2]: Exception raised from allgather_into_tensor_coalesced at /pytorch/torch/csrc/distributed/c10d/Backend.hpp:190 (most recent call first):
[rank2]: frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xd4 (0xffff95533ea4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank2]: frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe4 (0xffff954d3e44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank2]: frame #2: + 0x5d356f0 (0xffff9aef56f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #3: + 0x5d5f9e8 (0xffff9af1f9e8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #4: + 0x5d6e26c (0xffff9af2e26c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #5: + 0x811a24 (0xffff7b041a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank2]: frame #6: + 0x5d834f8 (0xffff9af434f8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #7: + 0x5d48730 (0xffff9af08730 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #8: + 0x5d48b88 (0xffff9af08b88 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #9: + 0x5d46ddc (0xffff9af06ddc in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #10: + 0x811a24 (0xffff7b041a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank2]: frame #11: + 0x5c29be4 (0xffff9ade9be4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #12: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) + 0x140 (0xffffa003bb50 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #13: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) + 0x1dc (0xffffa003bdac in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #14: + 0xb3cf44 (0xffff9ff3cf44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #15: + 0x588f5c (0xffff9f988f5c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #16: /root/miniconda3/bin/python3.11() [0x4d1e68]
[rank2]: frame #17: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank2]: frame #18: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank2]: frame #19: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #20: _PyObject_FastCallDictTstate + 0x7c (0x485b4c in /root/miniconda3/bin/python3.11)
[rank2]: frame #21: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank2]: frame #22: /root/miniconda3/bin/python3.11() [0x4f7c98]
[rank2]: frame #23: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank2]: frame #24: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank2]: frame #25: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #26: + 0x9e6610 (0xffff9fde6610 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #27: /root/miniconda3/bin/python3.11() [0x4d1e38]
[rank2]: frame #28: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank2]: frame #29: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank2]: frame #30: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #31: _PyObject_FastCallDictTstate + 0x100 (0x485bd0 in /root/miniconda3/bin/python3.11)
[rank2]: frame #32: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank2]: frame #33: /root/miniconda3/bin/python3.11() [0x4f7e20]
[rank2]: frame #34: /root/miniconda3/bin/python3.11() [0x4efed4]
[rank2]: frame #35: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank2]: frame #36: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank2]: frame #37: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #38: PyEval_EvalCode + 0xa8 (0x570928 in /root/miniconda3/bin/python3.11)
[rank2]: frame #39: /root/miniconda3/bin/python3.11() [0x5b7dfc]
[rank2]: frame #40: /root/miniconda3/bin/python3.11() [0x5b8144]
[rank2]: frame #41: /root/miniconda3/bin/python3.11() [0x5b8274]
[rank2]: frame #42: _PyRun_SimpleFileObject + 0x144 (0x5bb3b4 in /root/miniconda3/bin/python3.11)
[rank2]: frame #43: _PyRun_AnyFileObject + 0x9c (0x5bb990 in /root/miniconda3/bin/python3.11)
[rank2]: frame #44: Py_RunMain + 0x7cc (0x5dbd6c in /root/miniconda3/bin/python3.11)
[rank2]: frame #45: Py_BytesMain + 0x64 (0x5dc3f4 in /root/miniconda3/bin/python3.11)
[rank2]: frame #46: + 0x276c4 (0xffffa15f76c4 in /lib64/libc.so.6)
[rank2]: frame #47: __libc_start_main + 0x98 (0xffffa15f77a8 in /lib64/libc.so.6)
[rank2]: frame #48: /root/miniconda3/bin/python3.11() [0x432440]
[ERROR] 2026-05-12-12:20:14 (PID:422, Device:4, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[ERROR] 2026-05-12-12:20:14 (PID:421, Device:3, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[ERROR] 2026-05-12-12:20:14 (PID:420, Device:2, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[rank6]: Traceback (most recent call last):
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 335, in
[rank6]: trainer = Trainer(args=args)
[rank6]: ^^^^^^^^^^^^^^^^^^
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 71, in init
[rank6]: self.model = self.get_model(model_provider)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 179, in get_model
[rank6]: self.lora_weight_manager.verify_lora_weights()
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 371, in verify_lora_weights
[rank6]: param_data = self._gather_dtensor(param.data) if self._is_dtensor(param.data) else param.data
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 84, in _gather_dtensor
[rank6]: full_tensor = param_data.full_tensor()
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 570, in full_tensor
[rank6]: redist_res = self.redistribute(
[rank6]: ^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 542, in redistribute
[rank6]: return Redistribute.apply(self, device_mesh, placements, async_op)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
[rank6]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 306, in forward
[rank6]: output = redistribute_local_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 213, in redistribute_local_tensor
[rank6]: new_local_tensor = current_placement._to_replicate_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 260, in _to_replicate_tensor
[rank6]: result = funcol.all_gather_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/_functional_collectives.py", line 205, in all_gather_tensor
[rank6]: tensor = torch.ops._c10d_functional.all_gather_into_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
[rank6]: return self._op(*args, (kwargs or {}))
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
[rank6]: Exception raised from allgather_into_tensor_coalesced at /pytorch/torch/csrc/distributed/c10d/Backend.hpp:190 (most recent call first):
[rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xd4 (0xffffa6943ea4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank6]: frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe4 (0xffffa68e3e44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank6]: frame #2: + 0x5d356f0 (0xffffac3056f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #3: + 0x5d5f9e8 (0xffffac32f9e8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #4: + 0x5d6e26c (0xffffac33e26c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #5: + 0x811a24 (0xffff8c451a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank6]: frame #6: + 0x5d834f8 (0xffffac3534f8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #7: + 0x5d48730 (0xffffac318730 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #8: + 0x5d48b88 (0xffffac318b88 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #9: + 0x5d46ddc (0xffffac316ddc in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #10: + 0x811a24 (0xffff8c451a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank6]: frame #11: + 0x5c29be4 (0xffffac1f9be4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #12: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) + 0x140 (0xffffb144bb50 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #13: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) + 0x1dc (0xffffb144bdac in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #14: + 0xb3cf44 (0xffffb134cf44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #15: + 0x588f5c (0xffffb0d98f5c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #16: /root/miniconda3/bin/python3.11() [0x4d1e68]
[rank6]: frame #17: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank6]: frame #18: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank6]: frame #19: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #20: _PyObject_FastCallDictTstate + 0x7c (0x485b4c in /root/miniconda3/bin/python3.11)
[rank6]: frame #21: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank6]: frame #22: /root/miniconda3/bin/python3.11() [0x4f7c98]
[rank6]: frame #23: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank6]: frame #24: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank6]: frame #25: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #26: + 0x9e6610 (0xffffb11f6610 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #27: /root/miniconda3/bin/python3.11() [0x4d1e38]
[rank6]: frame #28: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank6]: frame #29: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank6]: frame #30: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #31: _PyObject_FastCallDictTstate + 0x100 (0x485bd0 in /root/miniconda3/bin/python3.11)
[rank6]: frame #32: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank6]: frame #33: /root/miniconda3/bin/python3.11() [0x4f7e20]
[rank6]: frame #34: /root/miniconda3/bin/python3.11() [0x4efed4]
[rank6]: frame #35: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank6]: frame #36: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank6]: frame #37: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #38: PyEval_EvalCode + 0xa8 (0x570928 in /root/miniconda3/bin/python3.11)
[rank6]: frame #39: /root/miniconda3/bin/python3.11() [0x5b7dfc]
[rank6]: frame #40: /root/miniconda3/bin/python3.11() [0x5b8144]
[rank6]: frame #41: /root/miniconda3/bin/python3.11() [0x5b8274]
[rank6]: frame #42: _PyRun_SimpleFileObject + 0x144 (0x5bb3b4 in /root/miniconda3/bin/python3.11)
[rank6]: frame #43: _PyRun_AnyFileObject + 0x9c (0x5bb990 in /root/miniconda3/bin/python3.11)
[rank6]: frame #44: Py_RunMain + 0x7cc (0x5dbd6c in /root/miniconda3/bin/python3.11)
[rank6]: frame #45: Py_BytesMain + 0x64 (0x5dc3f4 in /root/miniconda3/bin/python3.11)
[rank6]: frame #46: + 0x276c4 (0xffffb2a076c4 in /lib64/libc.so.6)
[rank6]: frame #47: __libc_start_main + 0x98 (0xffffb2a077a8 in /lib64/libc.so.6)
[rank6]: frame #48: /root/miniconda3/bin/python3.11() [0x432440]
[rank0]: Traceback (most recent call last):
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 335, in
[rank0]: trainer = Trainer(args=args)
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 71, in init
[rank0]: self.model = self.get_model(model_provider)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 179, in get_model
[rank0]: self.lora_weight_manager.verify_lora_weights()
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 371, in verify_lora_weights
[rank0]: param_data = self._gather_dtensor(param.data) if self._is_dtensor(param.data) else param.data
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 84, in _gather_dtensor
[rank0]: full_tensor = param_data.full_tensor()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 570, in full_tensor
[rank0]: redist_res = self.redistribute(
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 542, in redistribute
[rank0]: return Redistribute.apply(self, device_mesh, placements, async_op)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
[rank0]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 306, in forward
[rank0]: output = redistribute_local_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 213, in redistribute_local_tensor
[rank0]: new_local_tensor = current_placement._to_replicate_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 260, in _to_replicate_tensor
[rank0]: result = funcol.all_gather_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/_functional_collectives.py", line 205, in all_gather_tensor
[rank0]: tensor = torch.ops._c10d_functional.all_gather_into_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
[rank0]: return self._op(*args, (kwargs or {}))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
[rank0]: Exception raised from allgather_into_tensor_coalesced at /pytorch/torch/csrc/distributed/c10d/Backend.hpp:190 (most recent call first):
[rank0]: frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xd4 (0xffff9d0f3ea4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank0]: frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe4 (0xffff9d093e44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank0]: frame #2: + 0x5d356f0 (0xffffa2ab56f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #3: + 0x5d5f9e8 (0xffffa2adf9e8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #4: + 0x5d6e26c (0xffffa2aee26c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #5: + 0x811a24 (0xffff82c01a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank0]: frame #6: + 0x5d834f8 (0xffffa2b034f8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #7: + 0x5d48730 (0xffffa2ac8730 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #8: + 0x5d48b88 (0xffffa2ac8b88 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #9: + 0x5d46ddc (0xffffa2ac6ddc in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #10: + 0x811a24 (0xffff82c01a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank0]: frame #11: + 0x5c29be4 (0xffffa29a9be4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #12: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) + 0x140 (0xffffa7bfbb50 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #13: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) + 0x1dc (0xffffa7bfbdac in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #14: + 0xb3cf44 (0xffffa7afcf44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #15: + 0x588f5c (0xffffa7548f5c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #16: /root/miniconda3/bin/python3.11() [0x4d1e68]
[rank0]: frame #17: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank0]: frame #18: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank0]: frame #19: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #20: _PyObject_FastCallDictTstate + 0x7c (0x485b4c in /root/miniconda3/bin/python3.11)
[rank0]: frame #21: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank0]: frame #22: /root/miniconda3/bin/python3.11() [0x4f7c98]
[rank0]: frame #23: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank0]: frame #24: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank0]: frame #25: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #26: + 0x9e6610 (0xffffa79a6610 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #27: /root/miniconda3/bin/python3.11() [0x4d1e38]
[rank0]: frame #28: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank0]: frame #29: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank0]: frame #30: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #31: _PyObject_FastCallDictTstate + 0x100 (0x485bd0 in /root/miniconda3/bin/python3.11)
[rank0]: frame #32: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank0]: frame #33: /root/miniconda3/bin/python3.11() [0x4f7e20]
[rank0]: frame #34: /root/miniconda3/bin/python3.11() [0x4efed4]
[rank0]: frame #35: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank0]: frame #36: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank0]: frame #37: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #38: PyEval_EvalCode + 0xa8 (0x570928 in /root/miniconda3/bin/python3.11)
[rank0]: frame #39: /root/miniconda3/bin/python3.11() [0x5b7dfc]
[rank0]: frame #40: /root/miniconda3/bin/python3.11() [0x5b8144]
[rank0]: frame #41: /root/miniconda3/bin/python3.11() [0x5b8274]
[rank0]: frame #42: _PyRun_SimpleFileObject + 0x144 (0x5bb3b4 in /root/miniconda3/bin/python3.11)
[rank0]: frame #43: _PyRun_AnyFileObject + 0x9c (0x5bb990 in /root/miniconda3/bin/python3.11)
[rank0]: frame #44: Py_RunMain + 0x7cc (0x5dbd6c in /root/miniconda3/bin/python3.11)
[rank0]: frame #45: Py_BytesMain + 0x64 (0x5dc3f4 in /root/miniconda3/bin/python3.11)
[rank0]: frame #46: + 0x276c4 (0xffffa91b76c4 in /lib64/libc.so.6)
[rank0]: frame #47: __libc_start_main + 0x98 (0xffffa91b77a8 in /lib64/libc.so.6)
[rank0]: frame #48: /root/miniconda3/bin/python3.11() [0x432440]
[ERROR] 2026-05-12-12:20:16 (PID:424, Device:6, RankID:-1) ERR99999 UNKNOWN applicaiton exception
W0512 12:20:17.016000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 418 closing signal SIGTERM
W0512 12:20:17.018000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 419 closing signal SIGTERM
W0512 12:20:17.021000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 420 closing signal SIGTERM
W0512 12:20:17.022000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 421 closing signal SIGTERM
W0512 12:20:17.022000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 422 closing signal SIGTERM
W0512 12:20:17.022000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 423 closing signal SIGTERM
W0512 12:20:17.024000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 424 closing signal SIGTERM
E0512 12:20:20.809000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 7 (pid: 425) of binary: /root/miniconda3/bin/python3.11
Traceback (most recent call last):
File "/root/miniconda3/bin/torchrun", line 7, in
sys.exit(main())
^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in main
run(args)
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
mindspeed_mm/fsdp/train/trainer.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2026-05-12_12:20:17
host : atomgit-0211-9
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 425)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
awk: cmd. line:1: BEGIN{printf "%.3f\n", *1000/}
awk: cmd. line:1: ^ syntax error
Elapsed Time Per iteration (ms):
Average Samples per Second:
mindspeed-MM qwen3.5-4b lora 微调报错：
错误：
核心问题不是前面的 warning，而是 LoRA 权重校验阶段触发了 HCCL 不支持的通信算子。
真正报错：RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
触发位置：mindspeed_mm/fsdp/utils/lora_weight_manager.py 的 verify_lora_weights()
它对 DTensor 调用了 full_tensor()，底层会走 all-gather。
在你当前 NPU + HCCL 环境下，这条 all-gather 路径（allgather_into_tensor_coalesced）不支持，所以多卡都在初始化模型后一起崩了。
最后的 awk syntax error 是连带现象：训练提前失败后，统计脚本拿不到迭代时间变量，才报这个错，不是主因。
前面这些基本都不是致命：
OMP_NUM_THREADS=1：性能提示
pkg_resources is deprecated：依赖告警
fast path is not available：会回退到 torch 实现，可跑但可能慢
环境：npu910b 8X32GB
npu-dirver 25.5.1
can 8.5.0
docker-image: swr.cn-south-1.myhuaweicloud.com/ascendhub--mindspeed-mm:2025.rc4-a2-arm
model: qwen3.5-4b
python 环境：
accelerate 1.2.0
gpytorch 1.14.3
mindspeed 0.12.1 /data/work/git/MindSpeed
mindspeed-mm 0.1 /data/work/git/MindSpeed-MM
numpy 1.26.0
torch 2.7.1
torch_npu 2.7.1.post2
torchaudio 2.7.1
torchdata 0.11.0
torchvision 0.22.1
transformers 5.2.0
verl_npu 0.0.1 /data/work/git/MindSpeed-MM/verl_plugin
任务：mindspeed-MM qwen3.5-4b Lora 微调
使用 examples/qwen3_5/finetune_qwen3_5_4B.sh
examples/qwen3_5/qwen3_5_4B_config.yaml 内容
并行策略
parallel:
fully_shard_parallel_size: auto
fsdp_plan:
apply_modules: # 如果要开prefetch的话，请不要随意修改apply_modules的顺序
- model.visual
- model.visual.blocks.{}
- model.language_model
- model.language_model.embed_tokens
- model.language_model.layers.{}
- lm_head
param_dtype: bf16
reduce_dtype: fp32
recompute: true
recompute_plan:
apply_modules:
- model.language_model.layers.{*}
ulysses_parallel_size: 1 # 开启 ulysses-cp 时, 请将 model 的 attn_implementation 设置为 flash_attention_2
数据相关配置
data:
dataset_param:
dataset_type: huggingface
#数据集属性
attr:
images: images
messages: messages
role_tag: role
content_tag: content
user_tag: user
assistant_tag: assistant
数据加载
dataloader_param:
pin_memory: true
shuffle: false
dataloader_mode: sampler
drop_last: true
sampler_type: BaseRandomBatchSampler
num_workers: 8
collate_param:
model_name: qwen3vl
ignore_pad_token_for_loss: true
模型配置
model:
model_id: qwen3_5
model_name_or_path: *HF_MODEL_LOAD_PATH
trust_remote_code: true
attn_implementation: sdpa
freeze:
- model.visual
loss 配置
loss_cfg:
loss_type: default # If you want raw loss in model, loss_type can be set to "raw".
router_aux_loss_coef: 0.0
chunkloss配置
enable_chunk_loss: true
chunkloss_plan:
apply_module: lm_head
chunk_size: 1024
activation offload 配置
enable_activation_offload: false
activation_offload_plan:
apply_modules:
- model.visual.blocks.{}
- model.language_model.layers.{}
融合算子配置
use_triton_gdn: true
训练配置
training:
micro_batch_size: 1
gradient_accumulation_steps: 1
seed: 42
lr: 1.0e-4
lr_decay_style: cosine
lr_warmup_ratio: 0.1
weight_decay: 0
train_iters: 10000
clip_grad: 0.0
init_model_with_meta_device: true
optimizer: adamw
adam_fused: true
save_interval: 10000
no_load_optim: true # Do not load optimizer state; remove if loading is needed.
no_load_rng: true # Do not load RNG state; remove if loading is needed.
no_save_optim: true # Do not save optimizer state; remove if saving is needed.
no_save_rng: true # Do not save RNG state; remove if saving is needed.
LoRA：仅训练低秩适配器，需安装 peft；续训可设置 pretrained_lora_path
lora:
enable: true
rank: 8
alpha: 16
dropout: 0.0
init_lora_weights: true
pretrained_lora_path: null
target_modules:
- model.language_model.layers.{}.self_attn.q_proj
- model.language_model.layers.{}.self_attn.k_proj
- model.language_model.layers.{}.self_attn.v_proj
- model.language_model.layers.{}.self_attn.o_proj
- model.language_model.layers.{}.mlp.gate_proj
- model.language_model.layers.{}.mlp.up_proj
- model.language_model.layers.{*}.mlp.down_proj
load: ./ckpt/hf_path/Qwen35-xxB-dcp # 替换为转换后的dcp权重
save: ./save_path
use_deter_comp: false
plugin:
- mindspeed_mm/fsdp/models/qwen3_5
- mindspeed_mm/fsdp/data/datasets/huggingface
工具配置
tools:
profile:
enable: false
profile_type: static
ranks: [0]
static_param:
level: level1
with_stack: false
with_memory: false
record_shapes: false
with_cpu: true
save_path: ./profiling
start_step: 10
end_step: 11
data_simplification: false
aic_metrics_type: PipeUtilization
memory_profile:
enable: false
start_step: 1
end_step: 2
save_path: ./memory_snapshot
dump_ranks: [0]
stacks: all
max_entries: null
mem_info: false
报错信息：
省略了一些日志
[rank2]: Traceback (most recent call last):
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 335, in
[rank2]: trainer = Trainer(args=args)
[rank2]: ^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 71, in init
[rank2]: self.model = self.get_model(model_provider)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 179, in get_model
[rank2]: self.lora_weight_manager.verify_lora_weights()
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 371, in verify_lora_weights
[rank2]: param_data = self._gather_dtensor(param.data) if self._is_dtensor(param.data) else param.data
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 84, in _gather_dtensor
[rank2]: full_tensor = param_data.full_tensor()
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 570, in full_tensor
[rank2]: redist_res = self.redistribute(
[rank2]: ^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 542, in redistribute
[rank2]: return Redistribute.apply(self, device_mesh, placements, async_op)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
[rank2]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 306, in forward
[rank2]: output = redistribute_local_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 213, in redistribute_local_tensor
[rank2]: new_local_tensor = current_placement._to_replicate_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 260, in _to_replicate_tensor
[rank2]: result = funcol.all_gather_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/_functional_collectives.py", line 205, in all_gather_tensor
[rank2]: tensor = torch.ops._c10d_functional.all_gather_into_tensor(
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/root/miniconda3/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
[rank2]: return self._op(*args, (kwargs or {}))
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
[rank2]: Exception raised from allgather_into_tensor_coalesced at /pytorch/torch/csrc/distributed/c10d/Backend.hpp:190 (most recent call first):
[rank2]: frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xd4 (0xffff95533ea4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank2]: frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe4 (0xffff954d3e44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank2]: frame #2: + 0x5d356f0 (0xffff9aef56f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #3: + 0x5d5f9e8 (0xffff9af1f9e8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #4: + 0x5d6e26c (0xffff9af2e26c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #5: + 0x811a24 (0xffff7b041a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank2]: frame #6: + 0x5d834f8 (0xffff9af434f8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #7: + 0x5d48730 (0xffff9af08730 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #8: + 0x5d48b88 (0xffff9af08b88 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #9: + 0x5d46ddc (0xffff9af06ddc in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #10: + 0x811a24 (0xffff7b041a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank2]: frame #11: + 0x5c29be4 (0xffff9ade9be4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank2]: frame #12: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) + 0x140 (0xffffa003bb50 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #13: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) + 0x1dc (0xffffa003bdac in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #14: + 0xb3cf44 (0xffff9ff3cf44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #15: + 0x588f5c (0xffff9f988f5c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #16: /root/miniconda3/bin/python3.11() [0x4d1e68]
[rank2]: frame #17: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank2]: frame #18: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank2]: frame #19: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #20: _PyObject_FastCallDictTstate + 0x7c (0x485b4c in /root/miniconda3/bin/python3.11)
[rank2]: frame #21: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank2]: frame #22: /root/miniconda3/bin/python3.11() [0x4f7c98]
[rank2]: frame #23: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank2]: frame #24: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank2]: frame #25: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #26: + 0x9e6610 (0xffff9fde6610 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank2]: frame #27: /root/miniconda3/bin/python3.11() [0x4d1e38]
[rank2]: frame #28: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank2]: frame #29: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank2]: frame #30: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #31: _PyObject_FastCallDictTstate + 0x100 (0x485bd0 in /root/miniconda3/bin/python3.11)
[rank2]: frame #32: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank2]: frame #33: /root/miniconda3/bin/python3.11() [0x4f7e20]
[rank2]: frame #34: /root/miniconda3/bin/python3.11() [0x4efed4]
[rank2]: frame #35: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank2]: frame #36: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank2]: frame #37: /root/miniconda3/bin/python3.11() [0x570874]
[rank2]: frame #38: PyEval_EvalCode + 0xa8 (0x570928 in /root/miniconda3/bin/python3.11)
[rank2]: frame #39: /root/miniconda3/bin/python3.11() [0x5b7dfc]
[rank2]: frame #40: /root/miniconda3/bin/python3.11() [0x5b8144]
[rank2]: frame #41: /root/miniconda3/bin/python3.11() [0x5b8274]
[rank2]: frame #42: _PyRun_SimpleFileObject + 0x144 (0x5bb3b4 in /root/miniconda3/bin/python3.11)
[rank2]: frame #43: _PyRun_AnyFileObject + 0x9c (0x5bb990 in /root/miniconda3/bin/python3.11)
[rank2]: frame #44: Py_RunMain + 0x7cc (0x5dbd6c in /root/miniconda3/bin/python3.11)
[rank2]: frame #45: Py_BytesMain + 0x64 (0x5dc3f4 in /root/miniconda3/bin/python3.11)
[rank2]: frame #46: + 0x276c4 (0xffffa15f76c4 in /lib64/libc.so.6)
[rank2]: frame #47: __libc_start_main + 0x98 (0xffffa15f77a8 in /lib64/libc.so.6)
[rank2]: frame #48: /root/miniconda3/bin/python3.11() [0x432440]
[ERROR] 2026-05-12-12:20:14 (PID:422, Device:4, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[ERROR] 2026-05-12-12:20:14 (PID:421, Device:3, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[ERROR] 2026-05-12-12:20:14 (PID:420, Device:2, RankID:-1) ERR99999 UNKNOWN applicaiton exception
[rank6]: Traceback (most recent call last):
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 335, in
[rank6]: trainer = Trainer(args=args)
[rank6]: ^^^^^^^^^^^^^^^^^^
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 71, in init
[rank6]: self.model = self.get_model(model_provider)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 179, in get_model
[rank6]: self.lora_weight_manager.verify_lora_weights()
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 371, in verify_lora_weights
[rank6]: param_data = self._gather_dtensor(param.data) if self._is_dtensor(param.data) else param.data
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 84, in _gather_dtensor
[rank6]: full_tensor = param_data.full_tensor()
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 570, in full_tensor
[rank6]: redist_res = self.redistribute(
[rank6]: ^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 542, in redistribute
[rank6]: return Redistribute.apply(self, device_mesh, placements, async_op)
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
[rank6]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 306, in forward
[rank6]: output = redistribute_local_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 213, in redistribute_local_tensor
[rank6]: new_local_tensor = current_placement._to_replicate_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 260, in _to_replicate_tensor
[rank6]: result = funcol.all_gather_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/_functional_collectives.py", line 205, in all_gather_tensor
[rank6]: tensor = torch.ops._c10d_functional.all_gather_into_tensor(
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: File "/root/miniconda3/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
[rank6]: return self._op(*args, (kwargs or {}))
[rank6]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
[rank6]: Exception raised from allgather_into_tensor_coalesced at /pytorch/torch/csrc/distributed/c10d/Backend.hpp:190 (most recent call first):
[rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xd4 (0xffffa6943ea4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank6]: frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe4 (0xffffa68e3e44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank6]: frame #2: + 0x5d356f0 (0xffffac3056f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #3: + 0x5d5f9e8 (0xffffac32f9e8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #4: + 0x5d6e26c (0xffffac33e26c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #5: + 0x811a24 (0xffff8c451a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank6]: frame #6: + 0x5d834f8 (0xffffac3534f8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #7: + 0x5d48730 (0xffffac318730 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #8: + 0x5d48b88 (0xffffac318b88 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #9: + 0x5d46ddc (0xffffac316ddc in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #10: + 0x811a24 (0xffff8c451a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank6]: frame #11: + 0x5c29be4 (0xffffac1f9be4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank6]: frame #12: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) + 0x140 (0xffffb144bb50 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #13: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) + 0x1dc (0xffffb144bdac in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #14: + 0xb3cf44 (0xffffb134cf44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #15: + 0x588f5c (0xffffb0d98f5c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #16: /root/miniconda3/bin/python3.11() [0x4d1e68]
[rank6]: frame #17: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank6]: frame #18: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank6]: frame #19: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #20: _PyObject_FastCallDictTstate + 0x7c (0x485b4c in /root/miniconda3/bin/python3.11)
[rank6]: frame #21: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank6]: frame #22: /root/miniconda3/bin/python3.11() [0x4f7c98]
[rank6]: frame #23: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank6]: frame #24: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank6]: frame #25: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #26: + 0x9e6610 (0xffffb11f6610 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank6]: frame #27: /root/miniconda3/bin/python3.11() [0x4d1e38]
[rank6]: frame #28: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank6]: frame #29: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank6]: frame #30: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #31: _PyObject_FastCallDictTstate + 0x100 (0x485bd0 in /root/miniconda3/bin/python3.11)
[rank6]: frame #32: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank6]: frame #33: /root/miniconda3/bin/python3.11() [0x4f7e20]
[rank6]: frame #34: /root/miniconda3/bin/python3.11() [0x4efed4]
[rank6]: frame #35: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank6]: frame #36: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank6]: frame #37: /root/miniconda3/bin/python3.11() [0x570874]
[rank6]: frame #38: PyEval_EvalCode + 0xa8 (0x570928 in /root/miniconda3/bin/python3.11)
[rank6]: frame #39: /root/miniconda3/bin/python3.11() [0x5b7dfc]
[rank6]: frame #40: /root/miniconda3/bin/python3.11() [0x5b8144]
[rank6]: frame #41: /root/miniconda3/bin/python3.11() [0x5b8274]
[rank6]: frame #42: _PyRun_SimpleFileObject + 0x144 (0x5bb3b4 in /root/miniconda3/bin/python3.11)
[rank6]: frame #43: _PyRun_AnyFileObject + 0x9c (0x5bb990 in /root/miniconda3/bin/python3.11)
[rank6]: frame #44: Py_RunMain + 0x7cc (0x5dbd6c in /root/miniconda3/bin/python3.11)
[rank6]: frame #45: Py_BytesMain + 0x64 (0x5dc3f4 in /root/miniconda3/bin/python3.11)
[rank6]: frame #46: + 0x276c4 (0xffffb2a076c4 in /lib64/libc.so.6)
[rank6]: frame #47: __libc_start_main + 0x98 (0xffffb2a077a8 in /lib64/libc.so.6)
[rank6]: frame #48: /root/miniconda3/bin/python3.11() [0x432440]
[rank0]: Traceback (most recent call last):
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 335, in
[rank0]: trainer = Trainer(args=args)
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 71, in init
[rank0]: self.model = self.get_model(model_provider)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/train/trainer.py", line 179, in get_model
[rank0]: self.lora_weight_manager.verify_lora_weights()
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 371, in verify_lora_weights
[rank0]: param_data = self._gather_dtensor(param.data) if self._is_dtensor(param.data) else param.data
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/work/git/MindSpeed-MM/mindspeed_mm/fsdp/utils/lora_weight_manager.py", line 84, in _gather_dtensor
[rank0]: full_tensor = param_data.full_tensor()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 570, in full_tensor
[rank0]: redist_res = self.redistribute(
[rank0]: ^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_api.py", line 542, in redistribute
[rank0]: return Redistribute.apply(self, device_mesh, placements, async_op)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
[rank0]: return super().apply(*args, **kwargs) # type: ignore[misc]
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 306, in forward
[rank0]: output = redistribute_local_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/_redistribute.py", line 213, in redistribute_local_tensor
[rank0]: new_local_tensor = current_placement._to_replicate_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/tensor/placement_types.py", line 260, in _to_replicate_tensor
[rank0]: result = funcol.all_gather_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/_functional_collectives.py", line 205, in all_gather_tensor
[rank0]: tensor = torch.ops._c10d_functional.all_gather_into_tensor(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/root/miniconda3/lib/python3.11/site-packages/torch/_ops.py", line 1158, in call
[rank0]: return self._op(*args, (kwargs or {}))
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: Backend hccl does not support allgather_into_tensor_coalesced
[rank0]: Exception raised from allgather_into_tensor_coalesced at /pytorch/torch/csrc/distributed/c10d/Backend.hpp:190 (most recent call first):
[rank0]: frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0xd4 (0xffff9d0f3ea4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank0]: frame #1: c10::detail::torchCheckFail(char const, char const, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe4 (0xffff9d093e44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libc10.so)
[rank0]: frame #2: + 0x5d356f0 (0xffffa2ab56f0 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #3: + 0x5d5f9e8 (0xffffa2adf9e8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #4: + 0x5d6e26c (0xffffa2aee26c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #5: + 0x811a24 (0xffff82c01a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank0]: frame #6: + 0x5d834f8 (0xffffa2b034f8 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #7: + 0x5d48730 (0xffffa2ac8730 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #8: + 0x5d48b88 (0xffffa2ac8b88 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #9: + 0x5d46ddc (0xffffa2ac6ddc in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #10: + 0x811a24 (0xffff82c01a24 in /root/miniconda3/lib/python3.11/site-packages/torch_npu/lib/libtorch_npu.so)
[rank0]: frame #11: + 0x5c29be4 (0xffffa29a9be4 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
[rank0]: frame #12: torch::jit::invokeOperatorFromPython(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, pybind11::args const&, pybind11::kwargs const&, std::optionalc10::DispatchKey) + 0x140 (0xffffa7bfbb50 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #13: torch::jit::_get_operation_for_overload_or_packet(std::vector<std::shared_ptrtorch::jit::Operator, std::allocator<std::shared_ptrtorch::jit::Operator > > const&, c10::Symbol, pybind11::args const&, pybind11::kwargs const&, bool, std::optionalc10::DispatchKey) + 0x1dc (0xffffa7bfbdac in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #14: + 0xb3cf44 (0xffffa7afcf44 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #15: + 0x588f5c (0xffffa7548f5c in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #16: /root/miniconda3/bin/python3.11() [0x4d1e68]
[rank0]: frame #17: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank0]: frame #18: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank0]: frame #19: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #20: _PyObject_FastCallDictTstate + 0x7c (0x485b4c in /root/miniconda3/bin/python3.11)
[rank0]: frame #21: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank0]: frame #22: /root/miniconda3/bin/python3.11() [0x4f7c98]
[rank0]: frame #23: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank0]: frame #24: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank0]: frame #25: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #26: + 0x9e6610 (0xffffa79a6610 in /root/miniconda3/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
[rank0]: frame #27: /root/miniconda3/bin/python3.11() [0x4d1e38]
[rank0]: frame #28: _PyObject_Call + 0x68 (0x485728 in /root/miniconda3/bin/python3.11)
[rank0]: frame #29: _PyEval_EvalFrameDefault + 0x4b64 (0x42ad34 in /root/miniconda3/bin/python3.11)
[rank0]: frame #30: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #31: _PyObject_FastCallDictTstate + 0x100 (0x485bd0 in /root/miniconda3/bin/python3.11)
[rank0]: frame #32: _PyObject_Call_Prepend + 0x13c (0x485eb0 in /root/miniconda3/bin/python3.11)
[rank0]: frame #33: /root/miniconda3/bin/python3.11() [0x4f7e20]
[rank0]: frame #34: /root/miniconda3/bin/python3.11() [0x4efed4]
[rank0]: frame #35: _PyObject_MakeTpCall + 0x98 (0x48595c in /root/miniconda3/bin/python3.11)
[rank0]: frame #36: _PyEval_EvalFrameDefault + 0x31c8 (0x429398 in /root/miniconda3/bin/python3.11)
[rank0]: frame #37: /root/miniconda3/bin/python3.11() [0x570874]
[rank0]: frame #38: PyEval_EvalCode + 0xa8 (0x570928 in /root/miniconda3/bin/python3.11)
[rank0]: frame #39: /root/miniconda3/bin/python3.11() [0x5b7dfc]
[rank0]: frame #40: /root/miniconda3/bin/python3.11() [0x5b8144]
[rank0]: frame #41: /root/miniconda3/bin/python3.11() [0x5b8274]
[rank0]: frame #42: _PyRun_SimpleFileObject + 0x144 (0x5bb3b4 in /root/miniconda3/bin/python3.11)
[rank0]: frame #43: _PyRun_AnyFileObject + 0x9c (0x5bb990 in /root/miniconda3/bin/python3.11)
[rank0]: frame #44: Py_RunMain + 0x7cc (0x5dbd6c in /root/miniconda3/bin/python3.11)
[rank0]: frame #45: Py_BytesMain + 0x64 (0x5dc3f4 in /root/miniconda3/bin/python3.11)
[rank0]: frame #46: + 0x276c4 (0xffffa91b76c4 in /lib64/libc.so.6)
[rank0]: frame #47: __libc_start_main + 0x98 (0xffffa91b77a8 in /lib64/libc.so.6)
[rank0]: frame #48: /root/miniconda3/bin/python3.11() [0x432440]
[ERROR] 2026-05-12-12:20:16 (PID:424, Device:6, RankID:-1) ERR99999 UNKNOWN applicaiton exception
W0512 12:20:17.016000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 418 closing signal SIGTERM
W0512 12:20:17.018000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 419 closing signal SIGTERM
W0512 12:20:17.021000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 420 closing signal SIGTERM
W0512 12:20:17.022000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 421 closing signal SIGTERM
W0512 12:20:17.022000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 422 closing signal SIGTERM
W0512 12:20:17.022000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 423 closing signal SIGTERM
W0512 12:20:17.024000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 424 closing signal SIGTERM
E0512 12:20:20.809000 352 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 7 (pid: 425) of binary: /root/miniconda3/bin/python3.11
Traceback (most recent call last):
File "/root/miniconda3/bin/torchrun", line 7, in
sys.exit(main())
^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in main
run(args)
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 139, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
mindspeed_mm/fsdp/train/trainer.py FAILED
Failures:
<NO_OTHER_FAILURES>
Root Cause (first observed failure):
[0]:
time : 2026-05-12_12:20:17
host : atomgit-0211-9
rank : 7 (local_rank: 7)
exitcode : 1 (pid: 425)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
awk: cmd. line:1: BEGIN{printf "%.3f\n", *1000/}
awk: cmd. line:1: ^ syntax error
Elapsed Time Per iteration (ms):
Average Samples per Second: