RaceCLIP

Requirements

Run the following command to install the required packages:

pip install -r requirements.txt

Offline UMLS knowledge retrieval

The term definitions and associated relationships can be retrieved via the UMLS API shown as following script:

python retrieval.py

Retrieval augmented MLLM captioning

Please download MedGemma-1.5-4B before medical image recaptioning. The detailed usage of MedGemma is available via the official repository provided by Google.

python MLLM_recaptioning.py

Multi-text contrastive learning

Now you can start to fine-tune the model from pulicly available weights pretrained by OpenAI using multi-text contrastive loss:

python main.py

Dataset description

This framework recaptions ROCO dataset with the integration of expert knowledge from the medical knowledge base UMLS. Each image in the recaptioned dataset is paired with 4 various augmented captions. Details

Acknowledgement

The implementation of RaceCLIP is based on MedGemma, CLIP and UMLS. We thank the authors for their open-sourced code and encourage users to cite their works when applicable.