Self-Supervised Contrastive Learning

Deep learning final project: implementation and comparison of SimCLR, MoCo v2, and BYOL on CIFAR-10/100.

Setup

pip install -r requirements.txt

Training

Local dev (M4 Max, MPS — smoke test 2 epochs):

bash scripts/local_smoke_test.sh

Slurm H200 (full training):

# SimCLR: BS=4096, 400 epochs (~3-4 hours)
sbatch scripts/slurm_train.sh --config configs/simclr_cifar10.yaml

# MoCo v2: BS=1024, 400 epochs (~4 hours)
sbatch scripts/slurm_train.sh --config configs/moco_cifar10.yaml

# BYOL: BS=2048, 400 epochs (~5 hours)
sbatch scripts/slurm_train.sh --config configs/byol_cifar10.yaml

Evaluation

# Linear probe (top-1, top-5)
python linear_eval.py --checkpoint runs/simclr_cifar10_bs2048/checkpoint_ep0400.pth

# k-NN (k=200, T=0.07)
python knn_eval.py --checkpoint runs/simclr_cifar10_bs2048/checkpoint_ep0400.pth

# Supervised baseline (oracle upper bound)
python supervised_baseline.py --dataset cifar10 --epochs 200

Visualization

# t-SNE comparison
python visualize.py --mode tsne \
    --checkpoints \
        runs/simclr_cifar10_bs2048/checkpoint_ep0400.pth \
        runs/moco_cifar10_bs512/checkpoint_ep0400.pth \
        runs/byol_cifar10_bs1024/checkpoint_ep0400.pth \
    --output_dir figures/

# Training loss curves
python visualize.py --mode loss_curves \
    --log_files \
        runs/simclr_cifar10_bs2048/train.log \
        runs/moco_cifar10_bs512/train.log \
        runs/byol_cifar10_bs1024/train.log \
    --output_dir figures/

# Augmentation pair examples
python visualize.py --mode aug_pairs --dataset cifar10 --output_dir figures/

Expected Results (ResNet-50, CIFAR-10, 400 epochs)

Method	Batch Size	Linear Probe Top-1	k-NN Top-1
Supervised	512	~95%	—
SimCLR	4096	~93%	~89%
MoCo v2	1024	~92%	~88%
BYOL	2048	~93–94%	~90%

Ablation Studies

The configs directory has CIFAR-10/100 variants. To run temperature ablations for SimCLR:

for T in 0.05 0.1 0.2 0.5; do
    python train.py --config configs/simclr_cifar10.yaml \
        --output_dir runs/ablation_temp_${T} \
        # edit temperature in config or add CLI override
done

Compute Notes

• Batch size: SimCLR performance scales with batch (more in-batch negatives). At BS=4096 on H200 (~5GB VRAM), this matches the original paper.
• num_workers: Set to $SLURM_CPUS_PER_TASK automatically on Slurm (default 8). Safe for a 64-core shared node.
• Mixed precision: BF16 on CUDA (H200), standard FP32 on MPS (M4 Max).