Data type not aligned

[KKKLeon]

Dear author, during training the data type of img embedding is float16, but the input type for llama is bf16, I am wondering if this misalignment leads to data precision loss during training? Since float16 is much more precise than bf16.

[RenShuhuai-Andy]

Hi, thanks for your interest.

I thinks it's ok to use fp16 for img embedding since it can reduce GPU memory, and it's a common practice for other MLLMs like video-llama. However, it would be possiable to improve performance by using bf16.