作者您好，我在运行您的代码时出现一下错误，请问如何解决？

[giraffeGYH]

lora_ga_batch_size=2,
lora_ga_direction=ArB2r,
lora_ga_iters=2,
lora_ga_max_length=1024,
lora_ga_scale=stable,
lora_ga_stable_gamma=16,
lora_modules=[],
lora_rank=8,
lorap_lr_ratio=None,
loss_scale=default,
loss_type=None,
lr_scheduler_kwargs=None,
lr_scheduler_type=cosine,
max_epochs=None,
max_grad_norm=1.0,
max_length=10240,
max_memory={},
max_model_len=None,
max_new_tokens=64,
max_pixels=None,
max_steps=-1,
metric=None,
metric_for_best_model=loss,
model=/data0/yhguo/paper5/EMIT/InternVL3-8B,
model_author=None,
model_kwargs={'MAX_NUM': 6, 'dynamic_image_size': True, 'neighbor_dis': 6},
model_name=None,
model_revision=None,
model_type=custom,
modules_to_save=[],
mp_parameters=,
neftune_noise_alpha=None,
new_special_tokens=[],
no_cuda=False,
norm_bbox=None,
num_beams=1,
num_labels=None,
num_train_epochs=2.0,
optim=adamw_torch,
optim_args=None,
optim_target_modules=None,
optimizer=None,
output_dir=/data0/yhguo/paper5/EMIT/stage1_outputs/v51-20260622-144600,
overwrite_output_dir=False,
packing=False,
packing_length=None,
padding_free=False,
padding_side=right,
past_index=-1,
per_device_eval_batch_size=1,
per_device_train_batch_size=4,
predict_with_generate=False,
prediction_loss_only=False,
problem_type=None,
push_to_hub=False,
push_to_hub_model_id=None,
push_to_hub_organization=None,
push_to_hub_token=<PUSH_TO_HUB_TOKEN>,
quant_bits=None,
quant_method=None,
ray_scope=last,
reft_args=None,
reft_intervention_type=LoreftIntervention,
reft_layer_key=None,
reft_layers=None,
reft_rank=4,
remove_unused_columns=True,
repetition_penalty=None,
report_to=['tensorboard'],
response_prefix=None,
restore_callback_states_from_checkpoint=False,
resume_from_checkpoint=None,
resume_only_model=False,
rope_scaling=None,
router_aux_loss_coef=0.0,
run_name=/data0/yhguo/paper5/EMIT/stage1_outputs/v51-20260622-144600,
save_on_each_node=False,
save_only_model=False,
save_safetensors=True,
save_steps=2000.0,
save_strategy=steps,
save_total_limit=1,
seed=42,
sequence_parallel_size=1,
shuffle_buffer_size=1000,
skip_memory_metrics=True,
sortish_sampler=False,
split_batches=None,
split_dataset_ratio=0.0,
stop_words=[],
stopping_strategy=first_exhausted,
stream=False,
streaming=False,
strict=False,
swanlab_exp_name=None,
swanlab_lark_secret=None,
swanlab_lark_webhook_url=None,
swanlab_mode=cloud,
swanlab_project=None,
swanlab_token=<SWANLAB_TOKEN>,
swanlab_workspace=None,
system=You are an industrial inspector who checks products by images.

eval_benchmark/evaluate_batch_mmad_choice_InternVl3-8B.py

/data0/yhguo/paper5/EMIT/InternVL3-8B

/data0/yhguo/paper5/EMIT/datasets/

python eval_benchmark/evaluate_batch_mmad_choice_InternVl3-8B.py --checkpoint /data0/yhguo/paper5/EMIT/InternVL3-8B --data-root /data0/yhguo/paper5/EMIT/datasets/,
target_modules=['all-linear'],
target_parameters=None,
target_regex=None,
task_type=causal_lm,
temperature=0.0,
template=custom,
template_backend=swift,
tf32=None,
top_k=None,
top_logprobs=None,
top_p=None,
torch_compile=False,
torch_compile_backend=None,
torch_compile_mode=None,
torch_dtype=torch.bfloat16,
torch_empty_cache_steps=None,
torchdynamo=None,
tpu_metrics_debug=False,
tpu_num_cores=None,
train_dataloader_shuffle=True,
train_type=full,
trainable_parameters=['projector', 'soft_prompt'],
trainable_parameters_regex=None,
truncation_strategy=delete,
tuner_backend=peft,
use_chat_template=True,
use_cpu=False,
use_dora=False,
use_flash_ckpt=False,
use_galore=False,
use_hf=False,
use_ipex=False,
use_legacy_prediction_loop=False,
use_liger_kernel=False,
use_logits_to_keep=None,
use_mps_device=False,
use_rslora=False,
use_swift_lora=False,
val_dataset=[],
val_dataset_shuffle=False,
vera_d_initial=0.1,
vera_dropout=0.0,
vera_projection_prng_key=0,
vera_rank=256,
vit_gradient_checkpointing=None,
vit_lr=None,
warmup_ratio=0.03,
warmup_steps=0,
weight_decay=0.0001,
zero_hpz_partition_size=None,
)
[INFO:swift] Loading the model using model_dir: /data0/yhguo/paper5/EMIT/InternVL3-8B
[INFO:swift] Using environment variable NEIGHBOR_DIS, Setting neighbor_dis: 6.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.01s/it]
Some weights of CustomizedInternVLChatModel were not initialized from the model checkpoint at /data0/yhguo/paper5/EMIT/InternVL3-8B and are newly initialized: ['projector.0.bias', 'projector.0.num_batches_tracked', 'projector.0.running_mean', 'projector.0.running_var', 'projector.0.weight', 'projector.1.bias', 'projector.1.weight', 'soft_prompt.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use mean_resizing=False
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00, 1.06s/it]
Some weights of CustomizedInternVLChatModel were not initialized from the model checkpoint at /data0/yhguo/paper5/EMIT/InternVL3-8B and are newly initialized: ['projector.0.bias', 'projector.0.num_batches_tracked', 'projector.0.running_mean', 'projector.0.running_var', 'projector.0.weight', 'projector.1.bias', 'projector.1.weight', 'soft_prompt.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use mean_resizing=False
The new embeddings will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use mean_resizing=False
[INFO:swift] model.hf_device_map: {'': device(type='cuda', index=0)}
[INFO:swift] model_info: ModelInfo(model_type='custom', model_dir='/data0/yhguo/paper5/EMIT/InternVL3-8B', torch_dtype=torch.bfloat16, max_model_len=32768, quant_method=None, quant_bits=None, rope_scaling={'factor': 2.0, 'rope_type': 'dynamic', 'type': 'dynamic'}, is_moe_model=False, config=InternVLChatConfig {
"_attn_implementation_autoset": true,
"_commit_hash": null,
"_name_or_path": "/data0/yhguo/paper5/EMIT/InternVL3-8B",
"architectures": [
"InternVLChatModel"
],
"auto_map": {
"AutoConfig": "configuration_internvl_chat.InternVLChatConfig",
"AutoModel": "modeling_internvl_chat.InternVLChatModel",
"AutoModelForCausalLM": "modeling_internvl_chat.InternVLChatModel"
},
"downsample_ratio": 0.5,
"dynamic_image_size": true,
"force_image_size": 448,
"hidden_size": 3584,
"image_fold": null,
"llm_config": {
"_attn_implementation_autoset": true,
"_name_or_path": "./pretrained/Qwen2.5-32B-Instruct",
"add_cross_attention": false,
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"attn_implementation": "eager",
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": 151643,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": 151643,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"hidden_act": "silu",
"hidden_size": 3584,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"initializer_range": 0.02,
"intermediate_size": 18944,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"length_penalty": 1.0,
"max_length": 20,
"max_position_embeddings": 32768,
"max_window_layers": 70,
"min_length": 0,
"model_type": "qwen2",
"moe_config": null,
"no_repeat_ngram_size": 0,
"num_attention_heads": 28,
"num_beam_groups": 1,
"num_beams": 1,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"num_return_sequences": 1,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": null,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"rms_norm_eps": 1e-06,
"rope_scaling": {
"factor": 2.0,
"rope_type": "dynamic",
"type": "dynamic"
},
"rope_theta": 1000000.0,
"sep_token_id": null,
"sliding_window": null,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": false,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torch_dtype": "bfloat16",
"torchscript": false,
"transformers_version": "4.49.0",
"typical_p": 1.0,
"use_bfloat16": true,
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 151675
},
"max_dynamic_patch": 12,
"min_dynamic_patch": 1,
"model_type": "internvl_chat",
"pad2square": false,
"ps_version": "v2",
"select_layer": -1,
"system_message": null,
"template": "internvl2_5",
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": null,
"use_backbone_lora": 0,
"use_llm_lora": 0,
"use_thumbnail": true,
"vision_config": {
"_attn_implementation_autoset": true,
"_name_or_path": "OpenGVLab/InternViT-6B-448px-V1-5",
"add_cross_attention": false,
"architectures": [
"InternVisionModel"
],
"attention_dropout": 0.0,
"auto_map": {
"AutoConfig": "configuration_intern_vit.InternVisionConfig",
"AutoModel": "modeling_intern_vit.InternVisionModel"
},
"bad_words_ids": null,
"begin_suppress_tokens": null,
"bos_token_id": null,
"capacity_factor": 1.2,
"chunk_size_feed_forward": 0,
"cross_attention_hidden_size": null,
"decoder_start_token_id": null,
"diversity_penalty": 0.0,
"do_sample": false,
"drop_path_rate": 0.1,
"dropout": 0.0,
"early_stopping": false,
"encoder_no_repeat_ngram_size": 0,
"eos_token_id": null,
"eval_capacity_factor": 1.4,
"exponential_decay_length_penalty": null,
"finetuning_task": null,
"forced_bos_token_id": null,
"forced_eos_token_id": null,
"hidden_act": "gelu",
"hidden_size": 1024,
"id2label": {
"0": "LABEL_0",
"1": "LABEL_1"
},
"image_size": 448,
"initializer_factor": 0.1,
"initializer_range": 1e-10,
"intermediate_size": 4096,
"is_decoder": false,
"is_encoder_decoder": false,
"label2id": {
"LABEL_0": 0,
"LABEL_1": 1
},
"laux_allreduce": "all_nodes",
"layer_norm_eps": 1e-06,
"length_penalty": 1.0,
"max_length": 20,
"min_length": 0,
"model_type": "intern_vit_6b",
"moe_coeff_ratio": 0.5,
"moe_intermediate_size": 768,
"moe_output_scale": 4.0,
"no_repeat_ngram_size": 0,
"noisy_gate_policy": "RSample_before",
"norm_type": "layer_norm",
"num_attention_heads": 16,
"num_beam_groups": 1,
"num_beams": 1,
"num_channels": 3,
"num_experts": 8,
"num_hidden_layers": 24,
"num_return_sequences": 1,
"num_routed_experts": 4,
"num_shared_experts": 4,
"output_attentions": false,
"output_hidden_states": false,
"output_scores": false,
"pad_token_id": null,
"patch_size": 14,
"prefix": null,
"problem_type": null,
"pruned_heads": {},
"qk_normalization": false,
"qkv_bias": true,
"remove_invalid_values": false,
"repetition_penalty": 1.0,
"return_dict": true,
"return_dict_in_generate": false,
"sep_token_id": null,
"shared_expert_intermediate_size": 3072,
"suppress_tokens": null,
"task_specific_params": null,
"temperature": 1.0,
"tf_legacy_loss": false,
"tie_encoder_decoder": false,
"tie_word_embeddings": true,
"tokenizer_class": null,
"top_k": 50,
"top_p": 1.0,
"torch_dtype": "bfloat16",
"torchscript": false,
"transformers_version": "4.49.0",
"typical_p": 1.0,
"use_bfloat16": true,
"use_flash_attn": false,
"use_moe": false,
"use_residual": true,
"use_rts": false,
"use_weighted_residual": false
}
}
, task_type='causal_lm', num_labels=None)
[INFO:swift] model.generation_config: GenerationConfig {
"eos_token_id": 151645,
"max_new_tokens": 64,
"pad_token_id": 151643
}

[INFO:swift] default_system: 'You are an industrial inspector who checks products by images.\n\neval_benchmark/evaluate_batch_mmad_choice_InternVl3-8B.py\n\n/data0/yhguo/paper5/EMIT/InternVL3-8B\n\n/data0/yhguo/paper5/EMIT/datasets/\n\n python eval_benchmark/evaluate_batch_mmad_choice_InternVl3-8B.py --checkpoint /data0/yhguo/paper5/EMIT/InternVL3-8B --data-root /data0/yhguo/paper5/EMIT/datasets/'
[INFO:swift] max_length: 10240
[INFO:swift] response_prefix: ''
[INFO:swift] agent_template: react_en
[INFO:swift] norm_bbox: norm1000
[INFO:swift] Start time of running main: 2026-06-22 14:46:05.797880
[INFO:swift] swift.version: 3.8.0.dev0
The new lm_head weights will be initialized from a multivariate normal distribution that has old embeddings' mean and covariance. As described in this article: https://nlp.stanford.edu/~johnhew/vocab-expansion.html. To disable this, use mean_resizing=False
/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via init_process_group or barrier . Using the current device set by the user.
warnings.warn( # warn only once
/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via init_process_group or barrier . Using the current device set by the user.
warnings.warn( # warn only once
[INFO:swift] train_dataset: Dataset({
features: ['messages', 'images'],
num_rows: 458
})
[INFO:swift] val_dataset: None
[INFO:swift] The TrainArguments will be saved in: /data0/yhguo/paper5/EMIT/stage1_outputs/v51-20260622-144600/args.json
[INFO:swift] model: CustomizedInternVLChatModel(
(vision_model): InternVisionModel(
(embeddings): InternVisionEmbeddings(
(patch_embedding): Conv2d(3, 1024, kernel_size=(14, 14), stride=(14, 14))
)
(encoder): InternVisionEncoder(
(layers): ModuleList(
(0): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): Identity()
(drop_path2): Identity()
)
(1): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.004)
(drop_path2): DropPath(drop_prob=0.004)
)
(2): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.009)
(drop_path2): DropPath(drop_prob=0.009)
)
(3): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.013)
(drop_path2): DropPath(drop_prob=0.013)
)
(4): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.017)
(drop_path2): DropPath(drop_prob=0.017)
)
(5): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.022)
(drop_path2): DropPath(drop_prob=0.022)
)
(6): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.026)
(drop_path2): DropPath(drop_prob=0.026)
)
(7): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.031)
(drop_path2): DropPath(drop_prob=0.031)
)
(8): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.035)
(drop_path2): DropPath(drop_prob=0.035)
)
(9): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.039)
(drop_path2): DropPath(drop_prob=0.039)
)
(10): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.044)
(drop_path2): DropPath(drop_prob=0.044)
)
(11): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.048)
(drop_path2): DropPath(drop_prob=0.048)
)
(12): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.052)
(drop_path2): DropPath(drop_prob=0.052)
)
(13): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.056)
(drop_path2): DropPath(drop_prob=0.056)
)
(14): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.061)
(drop_path2): DropPath(drop_prob=0.061)
)
(15): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.065)
(drop_path2): DropPath(drop_prob=0.065)
)
(16): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.069)
(drop_path2): DropPath(drop_prob=0.069)
)
(17): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.074)
(drop_path2): DropPath(drop_prob=0.074)
)
(18): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.078)
(drop_path2): DropPath(drop_prob=0.078)
)
(19): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.083)
(drop_path2): DropPath(drop_prob=0.083)
)
(20): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.087)
(drop_path2): DropPath(drop_prob=0.087)
)
(21): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.091)
(drop_path2): DropPath(drop_prob=0.091)
)
(22): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.096)
(drop_path2): DropPath(drop_prob=0.096)
)
(23): InternVisionEncoderLayer(
(attn): InternAttention(
(qkv): Linear(in_features=1024, out_features=3072, bias=True)
(attn_drop): Dropout(p=0.0, inplace=False)
(proj_drop): Dropout(p=0.0, inplace=False)
(proj): Linear(in_features=1024, out_features=1024, bias=True)
)
(mlp): InternMLP(
(act): GELUActivation()
(fc1): Linear(in_features=1024, out_features=4096, bias=True)
(fc2): Linear(in_features=4096, out_features=1024, bias=True)
)
(norm1): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(norm2): LayerNorm((1024,), eps=1e-06, elementwise_affine=True)
(drop_path1): DropPath(drop_prob=0.100)
(drop_path2): DropPath(drop_prob=0.100)
)
)
)
)
(language_model): Qwen2ForCausalLM(
(model): Qwen2Model(
(embed_tokens): Embedding(151675, 3584)
(layers): ModuleList(
(0-27): 28 x Qwen2DecoderLayer(
(self_attn): Qwen2Attention(
(q_proj): Linear(in_features=3584, out_features=3584, bias=True)
(k_proj): Linear(in_features=3584, out_features=512, bias=True)
(v_proj): Linear(in_features=3584, out_features=512, bias=True)
(o_proj): Linear(in_features=3584, out_features=3584, bias=False)
)
(mlp): Qwen2MLP(
(gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
(up_proj): Linear(in_features=3584, out_features=18944, bias=False)
(down_proj): Linear(in_features=18944, out_features=3584, bias=False)
(act_fn): SiLU()
)
(input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
(post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
)
)
(norm): Qwen2RMSNorm((3584,), eps=1e-06)
(rotary_emb): Qwen2RotaryEmbedding()
)
(lm_head): Linear(in_features=3584, out_features=151675, bias=False)
)
(mlp1): Sequential(
(0): LayerNorm((4096,), eps=1e-05, elementwise_affine=True)
(1): Linear(in_features=4096, out_features=3584, bias=True)
(2): GELU(approximate='none')
(3): Linear(in_features=3584, out_features=3584, bias=True)
)
(projector): Sequential(
(0): BatchNorm2d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
(1): Conv2d(1, 3584, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))
)
(soft_prompt): Embedding(9, 3584)
)
[INFO:swift] model_parameter_info: CustomizedInternVLChatModel: 7944.4741M Params (0.0932M Trainable [0.0012%]), 0.0001M Buffers.
/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/mixin.py:104: FutureWarning: tokenizer is deprecated and will be removed in version 5.0.0 for CustomizedSeq2SeqTrainer.__init__. Use processing_class instead.
super().init(
/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/mixin.py:104: FutureWarning: tokenizer is deprecated and will be removed in version 5.0.0 for CustomizedSeq2SeqTrainer.__init__. Use processing_class instead.
super().init(
[INFO:swift] use_reentrant: True
[INFO:swift] The logging file will be saved in: /data0/yhguo/paper5/EMIT/stage1_outputs/v51-20260622-144600/logging.jsonl
[INFO:swift] Successfully registered post_encode hook: ['CustomizedInternVLChatModel'].
[rank1]: Traceback (most recent call last):
[rank1]: File "/data0/yhguo/paper5/EMIT/swift_sft.py", line 124, in
[rank1]: sft_main()
[rank1]: File "/data0/yhguo/paper5/EMIT/swift_sft.py", line 121, in sft_main
[rank1]: return CustomizedSwiftSft(args).main()
[rank1]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/llm/base.py", line 49, in main
[rank1]: result = self.run()
[rank1]: File "/data0/yhguo/paper5/EMIT/swift_sft.py", line 117, in run
[rank1]: return self.train(trainer)
[rank1]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/llm/train/sft.py", line 235, in train
[rank1]: trainer.train(trainer.args.resume_from_checkpoint)
[rank1]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/mixin.py", line 676, in train
[rank1]: res = super().train(*args, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/transformers/trainer.py", line 2241, in train
[rank1]: return inner_training_loop(
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank1]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/trainers.py", line 413, in training_step
[rank1]: return super().training_step(model, inputs, *args, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/transformers/trainer.py", line 3698, in training_step
[rank1]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
[rank1]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/trainers.py", line 327, in compute_loss
[rank1]: outputs = model(**inputs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank1]: return forward_call(*args, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank1]: ret_val = func(*args, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1899, in forward
[rank1]: loss = self.module(*inputs, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank1]: return self._call_impl(*args, **kwargs)
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank1]: return inner()
[rank1]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank1]: result = forward_call(*args, **kwargs)
[rank1]: File "/data0/yhguo/paper5/EMIT/model/internvl_chat/modeling_internvl_chat.py", line 359, in forward
[rank1]: soft_prompt = self.soft_prompt(torch.arange(self.num_soft_prompt_tokens).to(input_ids.device))
[rank1]: AttributeError: 'NoneType' object has no attribute 'device'
Train: 0%| | 0/116 [00:00<?, ?it/s][INFO:swift] use_logits_to_keep: False
[INFO:swift] last_model_checkpoint: None
[INFO:swift] best_model_checkpoint: None
[INFO:swift] images_dir: /data0/yhguo/paper5/EMIT/stage1_outputs/v51-20260622-144600/images
[rank0]: Traceback (most recent call last):
[rank0]: File "/data0/yhguo/paper5/EMIT/swift_sft.py", line 124, in
[rank0]: sft_main()
[rank0]: File "/data0/yhguo/paper5/EMIT/swift_sft.py", line 121, in sft_main
[rank0]: return CustomizedSwiftSft(args).main()
[rank0]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/llm/base.py", line 49, in main
[rank0]: result = self.run()
[rank0]: File "/data0/yhguo/paper5/EMIT/swift_sft.py", line 117, in run
[rank0]: return self.train(trainer)
[rank0]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/llm/train/sft.py", line 235, in train
[rank0]: trainer.train(trainer.args.resume_from_checkpoint)
[rank0]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/mixin.py", line 676, in train
[rank0]: res = super().train(*args, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/transformers/trainer.py", line 2241, in train
[rank0]: return inner_training_loop(
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/trainers.py", line 413, in training_step
[rank0]: return super().training_step(model, inputs, *args, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/transformers/trainer.py", line 3698, in training_step
[rank0]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
[rank0]: File "/data0/yhguo/paper5/EMIT/ms-main/swift/trainers/trainers.py", line 327, in compute_loss
[rank0]: outputs = model(**inputs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]: ret_val = func(*args, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1899, in forward
[rank0]: loss = self.module(*inputs, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1857, in _call_impl
[rank0]: return inner()
[rank0]: File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1805, in inner
[rank0]: result = forward_call(*args, **kwargs)
[rank0]: File "/data0/yhguo/paper5/EMIT/model/internvl_chat/modeling_internvl_chat.py", line 359, in forward
[rank0]: soft_prompt = self.soft_prompt(torch.arange(self.num_soft_prompt_tokens).to(input_ids.device))
[rank0]: AttributeError: 'NoneType' object has no attribute 'device'
Train: 0%| | 0/116 [00:00<?, ?it/s]
[rank0]:[W622 14:46:14.869509546 ProcessGroupNCCL.cpp:1479] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
W0622 14:46:14.943000 3198847 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 3198923 closing signal SIGTERM
E0622 14:46:15.258000 3198847 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 1 (pid: 3198924) of binary: /home/yaohuaguo/miniconda3/envs/anomaly-r1/bin/python3.10
Traceback (most recent call last):
File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/bin/torchrun", line 8, in
sys.exit(main())
File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in main
run(args)
File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/run.py", line 883, in run
elastic_launch(
File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 139, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/yaohuaguo/miniconda3/envs/anomaly-r1/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

swift_sft.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2026-06-22_14:46:14
host : ubun
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 3198924)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html