training

Multi-Modal DR Training with OpenCLIP

We provide release code and a patch to OpenCLIP for training models on DR datasets (DataComp, DFN, or custom datasets generated with MobileCLIP-DR.

In additional to full offline training, we provide support for online knowledge distillation from an ensemble of CLIP teachers.

Data

DFNDR

Not yet available.

DataCompDR

Our reinforcements to DataComp are available on HuggingFace.

• DataCompDR-12M
• DataCompDR-12M-BFloat16
• DataCompDR-1B

Our data does not include the original images and captions. For DataCompDR-12M, there is a corresponding DataComp-12M with original captions. One needs to download both datasets, then run the following script to join them:

#!/bin/bash
DATACOMP12M_PATH="./datasets/DataComp-12M/" # Download path of DataComp-12M from HF
DATACOMPDR12M_NOIMG_PATH="./datasets/DataCompDR-12M-noimage/" # Download path of DataCompDR-12M from HF
DATACOMPDR12M_PATH="./datasets/DataCompDR-12M/"
for  i in {00000000..00001023}
do
  mkdir tmp
  tar -xf $DATACOMP12M_PATH/${i}.tar -C tmp
  tar -xf $DATACOMP12M_NOIMG_PATH/${i}.tar -C tmp
  tar -cf $DATACOMPDR12M_PATH/${i}.tar -C tmp *.*
  rm -rf tmp
done

The images have to be downloaded separately. See hf_dataset_example.py for an example of downloading a single image.

Installing dependencies

We use OpenCLIP for training. We have made minor modifications to OpenCLIP for support of loading reinforcements and the training loss. To checkout the specific version of each library and apply our corresponding patch run the following commands in order:

# Clone MobileCLIP repository
git clone git@github.com:apple/ml-mobileclip.git
cd ml-mobileclip/

# Clone OpenCLIP repository, apply patch, and install
git clone https://github.com/mlfoundations/open_clip.git
cd open_clip
git checkout 7260a46e7b4bcf518f5200fea06da5bc85aae025  # Mon Mar 17 18:18:30 2025 -0400
git apply ../open_clip_v2.patch
cp ../configs/ ./ -r
cp ../dr/ ./src/open_clip_train/ -r
cp ../../mobileclip2/* ./src/open_clip/ -r

We retain the v1 patch for reproducibility. One can reproduce v1 OpenCLIP code by following these steps:

# Revert changes for compatibility with older OpenCLIP commit
sed -i 's/open_clip_train/training/g' ../dr/transforms.py
find . --name "*.sh" | xargs sed -i 's/open_clip_train/training/g'

# Clone OpenCLIP repository, apply patch, and install
git clone https://github.com/mlfoundations/open_clip.git
cd open_clip
git checkout cf86ee7ec4658845f640858ecd34d0f15588271a  # Wed May 29 21:57:08 2024 +0700
git apply ../open_clip_v1.patch
cp ../configs/ ./ -r
cp ../dr/ ./src/training/ -r

Training

DFNDR

We provide scripts for training on DFNDR-2B12M and DFNDR-2B.

cd open_clip/
bash configs/run_dfndr12m.sh  # Train a MobileCILP-B on DFN-2B12M with DR
bash configs/run_dfndr1B.sh  # Train a MobileCLIP-B on DFN-2B with DR

DataCompDR

We provide scripts for training on DataCompDR-12M and DataCompDR-1B.

cd open_clip/
bash configs/run_datacomp12m.sh  # Train a ViT-B/16 on DataComp-12M without DR
bash configs/run_datacompdr12m.sh  # Train a ViT-B/16 on DataComp-12M with DR
bash configs/run_datacompdr1B.sh  # Train a ViT-B/16 on DataComp-1B with DR