Hello! Thank you for maintaining this excellent project. I encountered an issue during fine-tuning Qwen3VL-8B on the CameraBench dataset and would like to seek your advice.
I fine-tuned Qwen3VL-8B using the CameraBench training set, but the performance on the test set actually degraded significantly. Taking the binary classification task for camera motion detection as an example, here is the performance comparison:
Original Qwen3VL-8B (zero-shot):
{
"overall_average_precision": 0.3631299885892711,
"overall_roc_auc": 0.7821918221542317,
"mean_roc_auc": 0.7821918221542317
}Fine-tuned Qwen3VL-8B:
{
"overall_average_precision": 0.21155924731349668,
"overall_roc_auc": 0.6230854336085513,
"mean_roc_auc": 0.6230854336085513
}Performance degradation:
Average Precision: -41.8% (0.363 → 0.212)
ROC-AUC: -20.4% (0.782 → 0.623)
I suspect the issue may stem from my fine-tuning hyper-parameter configuration. I followed the original paper and Qwen's official fine-tuning guidelines, but may have made mistakes in certain settings. Could you please review the following configuration and advise on what might be wrong and how to adjust it?
Training Configuration
lr: 2e-5
batch_size: 4
grad_accum_steps: 2
num_train_epochs: 6.0
weight_decay: 0
warmup_ratio: 0.05
max_grad_norm: 1
lr_scheduler_type: cosine
model_max_length: 8192
# Video processing
video_max_frames: 32
video_min_frames: 8
video_max_pixels: 16384
video_fps: 8.0
# Training flags
--data_flatten True
--tune_mm_vision False # Freeze vision encoder
--tune_mm_mlp True # Only train MLP
--tune_mm_llm False # Freeze LLM backbone
--bf16
Training Curves
The W&B dashboard (see attached screenshot) shows stable training dynamics with smooth loss convergence and stable gradients. However, I did not log validation metrics during training.
Question: Which aspects of my training hyperparameters might be problematic? Specifically, should I adjust the learning rate, the MLP-only fine-tuning strategy, the number of epochs, or other settings? What modifications would you recommend to achieve the expected performance improvement?
I plan to conduct further experiments on both 8B and 32B parameter models to verify the root cause of this issue. If there are any additional experiments you'd like me to run to help diagnose this, please let me know.
Thank you very much for your help!
Hello! Thank you for maintaining this excellent project. I encountered an issue during fine-tuning Qwen3VL-8B on the CameraBench dataset and would like to seek your advice.
I fine-tuned Qwen3VL-8B using the CameraBench training set, but the performance on the test set actually degraded significantly. Taking the binary classification task for camera motion detection as an example, here is the performance comparison:
Original Qwen3VL-8B (zero-shot):
{ "overall_average_precision": 0.3631299885892711, "overall_roc_auc": 0.7821918221542317, "mean_roc_auc": 0.7821918221542317 }Fine-tuned Qwen3VL-8B:
{ "overall_average_precision": 0.21155924731349668, "overall_roc_auc": 0.6230854336085513, "mean_roc_auc": 0.6230854336085513 }Performance degradation:
Average Precision: -41.8% (0.363 → 0.212)
ROC-AUC: -20.4% (0.782 → 0.623)
I suspect the issue may stem from my fine-tuning hyper-parameter configuration. I followed the original paper and Qwen's official fine-tuning guidelines, but may have made mistakes in certain settings. Could you please review the following configuration and advise on what might be wrong and how to adjust it?
Training Configuration
Training Curves
The W&B dashboard (see attached screenshot) shows stable training dynamics with smooth loss convergence and stable gradients. However, I did not log validation metrics during training.
Question: Which aspects of my training hyperparameters might be problematic? Specifically, should I adjust the learning rate, the MLP-only fine-tuning strategy, the number of epochs, or other settings? What modifications would you recommend to achieve the expected performance improvement?
I plan to conduct further experiments on both 8B and 32B parameter models to verify the root cause of this issue. If there are any additional experiments you'd like me to run to help diagnose this, please let me know.
Thank you very much for your help!