Hi, thanks for your great work on proposing a novel mechanism of processing long videos.
I've tried to use the model on my dataset, but my GPU resources are limited. When I finetune the model with fsdp enabled, everything is fine and just a CUDA OOM error (which is fine in terms of the problem I'm asking in this post). But when I turn off fsdp and enable deepspeed with the following config:
{
"compute_environment": "LOCAL_MACHINE",
"debug": false,
"deepspeed_config": {
"gradient_accumulation_steps": 1,
"gradient_clipping": 8.0,
"offload_optimizer_device": "cpu",
"offload_param_device": "cpu",
"zero3_init_flag": false,
"zero3_save_16bit_model": false,
"zero_stage": 2
},
"zero_optimization": {
"stage": 2,
"offload_optimizer": {
"device": "cpu"
},
"overlap_comm": true,
"contiguous_gradients": true,
"reduce_bucket_size": 500000000,
"sub_group_size": 1000000000,
"gather_16bit_weights_on_model_save": true
},
"train_micro_batch_size_per_gpu": 1,
"distributed_type": "DEEPSPEED",
"downcast_bf16": "no",
"dynamo_config": {
"dynamo_backend": "INDUCTOR"
},
"enable_cpu_affinity": false,
"machine_rank": 0,
"main_training_function": "main",
"mixed_precision": "bf16",
"num_machines": 1,
"num_processes": 2,
"rdzv_backend": "static",
"same_network": true,
"tpu_env": [],
"tpu_use_cluster": false,
"tpu_use_sudo": false,
"use_cpu": false
}
I continuously encounter the errors that the model dtype is bfloat16 but the input data passed to the model is float16, causing errors like:
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16
I did try to manually transfer the input (such as hidden states, embeddings, etc.) to bfloat16 before the forward pass, but there are so many of them in the codebase. Furthermore, previously when I use fsdp, there's no such dtype mismatch.
So my question is, did you use deepspeed before for training the model? If yes, may I ask did you encounter similar errors? If no, I'd love to hear your opinions on using deepspeed in this project.
Thank you.
Hi, thanks for your great work on proposing a novel mechanism of processing long videos.
I've tried to use the model on my dataset, but my GPU resources are limited. When I finetune the model with fsdp enabled, everything is fine and just a CUDA OOM error (which is fine in terms of the problem I'm asking in this post). But when I turn off fsdp and enable deepspeed with the following config:
I continuously encounter the errors that the model dtype is bfloat16 but the input data passed to the model is float16, causing errors like:
RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16
I did try to manually transfer the input (such as hidden states, embeddings, etc.) to bfloat16 before the forward pass, but there are so many of them in the codebase. Furthermore, previously when I use fsdp, there's no such dtype mismatch.
So my question is, did you use deepspeed before for training the model? If yes, may I ask did you encounter similar errors? If no, I'd love to hear your opinions on using deepspeed in this project.
Thank you.