ViP

This is the official code for our MICCAI 2024 (Early Accepted, Oral) paper:

Aligning Medical Images with General Knowledge from Large Language Models
Xiao Fang*, Yi Lin*, Dong Zhang, Kwang-Ting Cheng, Hao Chen

Requirement

# Clone the following repository
git clone https://github.com/KaiyangZhou/Dassl.pytorch

# Install torch
torch==1.13.1+cu117 torchvision==0.14.1+cu117 torchaudio==0.13.1 --extra-index-url https://download.pytorch.org/whl/cu117

# Install dependencies
pip install -r requirments.txt

# Install Dassl library
cd Dassl.pytorch
python setup.py develop

Data preparation

Pneumonia: Please download data here. We use the chest xray part. The data should be put in the following structure:

|-- /DATA/Pneumonia/chest_xray
|  |-- train
|      |-- normal lung
|          |-- NORMAL-28501-0001.jpeg
|          |--...
|      |-- pneumonia
|          |-- BACTERIA-7422-0001.jpeg
|          |--...
|  |-- test
|      |-- normal lung
|          |-- NORMAL-4512-0001.jpeg
|          |--...
|      |-- pneumonia
|          |-- BACTERIA-40699-0001.jpeg
|          |--...

Derm7pt: Please download data here. We follow this paper to split the data. The data should be put in the following structure:

|-- /DATA/Derm7pt/image
|   |-- train
|       |-- melanoma
|           |-- Aal002bis.jpg
|           |--...
|       |-- nevus
|           |-- Aal012.jpg
|           |--...
|   |-- val
|       |-- melanoma
|           |-- Ael490.jpg
|           |--...
|       |-- nevus
|           |-- Aal004.jpg
|           |--...
|   |-- test
|       |-- melanoma
|           |-- Aal002.jpg
|           |--...
|       |-- nevus
|           |-- Aal008.jpg
|           |--...

We also provide the data split in the DATA folder.

Training & Evaluation

We provide the following prompt tuning methods:

Method	Paper	Configs	Training Scripts
ViP	link	link	link
CoOp	link	link	link
CoCoOp	link	link	link
MaPLe	link	link	link
VPT	link	link	link

We provide the use of shell scripts for training and evaluation using ViP as an example.
Pneumonia:

bash scripts/ViP/main_pneumonia.sh

Derm7pt:

bash scripts/ViP/main_derm.sh

Citation

Please cite the paper if you use the code.

@inproceedings{fang2024aligning,
  title={Aligning Medical Images with General Knowledge from Large Language Models},
  author={Fang, Xiao and Lin, Yi and Zhang, Dong and Cheng, Kwang-Ting and Chen, Hao},
  booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
  pages={57--67},
  year={2024},
  organization={Springer}
}

Acknowledgment

The code is built on CoOp, thanks for their amazing work!