How to reproduce the experimental results in the paper?

[wuhang03]

Hi there,

thank you so much for your excellent work and clean codes, this insightful dataset and benchmark contributes a lot to the community.

However, I met some problems when trying to reproduce the experimental results in the paper since this repository only provides inference scripts.

It'll be very helpful if you can upload the evaluation scripts or just give me some tips about it.

Thank you in advance for your help!

[linzhiqiu]

Thanks for reaching out. @chancharikmitra We are planning a major upgrade to the codebase to support seamless evaluation of VQA/retrieval. Feel free to also email us if you have any direct questions, but please expect an update to the codebase this week!

[wuhang03]

Glad to hear! Looking forward to your update.

[linzhiqiu]

Dear all,

Thank you for your patience, and apologies for the delay! We are happy to share the following resources for CameraBench:

Finetuning Code: https://github.com/chancharikmitra/cambench_ft

Evaluation Code: https://github.com/linzhiqiu/t2v_metrics/tree/main/camerabench

Let us know if you encounter any issues. We’re excited to see what you build with CameraBench!

Best,
CameraBench Team

[Tangshitao]

├─ Move_In : mAP = 0.7619, AUC = 0.8764 (542 samples, 158+/384-)
├─ Move_Left : mAP = 0.3872, AUC = 0.8210 (483 samples, 55+/428-)
├─ Move_Out : mAP = 0.3826, AUC = 0.8163 (542 samples, 52+/490-)
├─ Move_Up : mAP = 0.1500, AUC = 0.6981 (424 samples, 29+/395-)
├─ Pan_Left : mAP = 0.7477, AUC = 0.9068 (506 samples, 79+/427-)
├─ Pan_Right : mAP = 0.7393, AUC = 0.9180 (506 samples, 77+/429-)
├─ Roll_Counterclockwise : mAP = 0.3806, AUC = 0.8542 (464 samples, 47+/417-)
├─ Static : mAP = 0.3918, AUC = 0.7654 (958 samples, 92+/866-)
├─ Tilt_Up : mAP = 0.6318, AUC = 0.8832 (452 samples, 58+/394-)
├─ Zoom_In : mAP = 0.4323, AUC = 0.8111 (444 samples, 50+/394-)
└─ Zoom_Out : mAP = 0.5248, AUC = 0.8295 (446 samples, 46+/400-)
───────────────────────── Model Average: mAP = 0.5027, AUC = 0.8346 (5767 total samples)

I run the binary classification script, but the mAP is much lower than what reported in the paper. The mAP is only 0.5 in my run, but the paper shows 0.6 mAP. Wonder what causes the difference.

[yyfl2zju]

Hi！I am in the same situation.

[chancharikmitra]

Thank you both for following up on this. I will take a look and follow up in a few days!

[ddz16]

Me too, My mAP is 0.4969.