Questions about result comparisons

[TianBaoGe]

Dear authors, thank you for your great work! I have a few questions regarding your experiments:

1. For the supervised results, did you perform full-tuning on all three backbones? Are these backbones initialized with CLIP's weights? In my experiments, ResNet50 easily achieved over 90% accuracy on the Pneumonia test set with fine-tuning。
2. Regarding methods like CoOp, CoCoOp, and MaPLe, did you reproduce their results yourself?

[xiaofang007]

Sorry for the confusion.

1. I check our implementation and we just initialize the model randomly which means we do not initialize with pretrained encoder like CLIP. I apologize that initializing with pretrained encoders may boost the performance easily.
2. Yes, we run the methods to obtain the result because there is no result on the two datasets reported in their paper.

[TianBaoGe]

Thanks for you quick reply~! So, is the key point of this work is to achieve better classification performance by training additional components while keeping CLIP's image and text encoders frozen?

[xiaofang007]

Hi, in my opinion, I do not trust pretrained encoders that much because these pretrained encoders are largely pretrained on natural images and may create bias for learning medical image features. If you check original CLIP paper, there is also performance drop when directly performing inference on medical images. But pretrained models still learn general features like color, texture and other features pretty well that are shared across domains. Our intuition is that if we can find a way to guide the model to analyze based on common features, it might improve domain adaptation. Also freezing the CLIP weight preserves learned knowledge and is more efficient to train.